|Home | About | Journals | Submit | Contact Us | Français|
Assess proxy respondent effects on health care evaluations by Medicare beneficiaries.
110,215 respondents from the nationally representative 2001 CAHPS® Medicare Fee-for-Service Survey.
We compare the effects of both proxy respondents and proxy assistance (reading, writing, or translating) on 23 “objective” report items and four “subjective” global measures of health care experiences using propensity-score-weighted regression. We assess whether proxy effects differ among spouses, other relatives, or nonrelatives.
Proxy respondents provide less positive evaluations of beneficiary health care experiences than otherwise similar self-reporting beneficiaries for more subjective global ratings (average effect of 0.21 standard deviations); differences are smaller for relatively objective and specific report items. Proxy assistance differences are similar, but about half as large. Reports from spouse proxy respondents are more positive than those from other proxies and are similar to what would have been reported by the beneficiaries themselves. Standard regression techniques may overestimate proxy effects in this instance.
One should treat proxy responses to subjective ratings cautiously. Even seemingly innocuous reading, writing, and translation by proxies may influence answers. Spouses may be accurate proxies for the elderly in evaluations of health care.
Surveys of the recipients of health care are increasingly important means of assessing the care provided by health plans, hospitals, physicians, and other entities (Lied et al. 2003; Tai-Seale 2004; Darby, Hays, and Kletke 2005). At the same time, an aging population means that more patients are unable to answer surveys themselves. The use of proxy respondents provides a practical solution to survey nonresponse and missing data attributable to limitations in the ability of patients, beneficiaries, or nursing home residents to respond themselves. By asking representatives to respond on behalf of the patient, survey researchers need not omit the experiences of the least healthy and most vulnerable. For these reasons, most health care surveys allow the use of proxy respondents. Yet, questions inevitably arise about whether proxies give responses comparable to what might have been self-reported. One might ask whether there is systematic bias or substantial decrement in reliability from the use of proxy responses that erodes much of the apparent gains from reducing the selective omission of those needing assistance in responding.
Prior research on proxy response has largely focused on differences between self- and proxy responses reporting on the experiences of the same individual. Much of this work has taken place with small convenience samples of patients, often with severe health problems (e.g., end-of-life, schizophrenia, or lung cancer) (Kutner et al. 2006; Hoe et al. 2007; Wennman-Larsen et al. 2007). In such a design, any differences between the self- and proxy responses are because the proxy is reporting on the same experiences in a different way than self-respondents. While typically in poor health, targets in these studies are by definition able and willing to provide self-responses, and thus differ fundamentally from individuals who require proxy assistance. Those who are unable to respond for themselves are not represented in these prior studies and may differ in other important ways.
This prior research finds that proxies can both over- and underestimate morbidity and disability (Magaziner et al. 1988; Sneeuw et al. 1997; Shaw, McColl, and Bond 2000; Todorov and Kirchner 2000; Duncan et al. 2002; Tang and McCorkle 2002; Pickard et al. 2004) and other aspects of health-related quality of life (Hays et al. 1995; Andresen, Vahle, and Lollar 2001; Becchi et al. 2004; Higashi et al. 2005; Kutner et al. 2006; Hoe et al. 2007). Proxy reporting differences may be attributable to different cognitive and perceptual strategies to answering questions (Todorov and Kirchner 2000; Stineman et al. 2004; Lynn Snow et al. 2005). Proxy respondents rely on observable factors, such as counts or the presence or absence of a symptom (Lynn Snow et al. 2005), suggesting smaller discrepancies between self- and proxy-reports for objective or observable measures than for subjective measures (Whiteman and Green 1997; Todorov and Kirchner 2000; Sneeuw, Sprangers, and Aaronson 2002; Pickard et al. 2004; Stineman et al. 2004).
The relationship of the proxy to the intended respondent may also influence the accuracy of proxy responses. Some research suggests that spouses and proxies who live with the intended respondent provide responses that are closer to those of the intended respondent than do other family members (Shaw et al. 2000; The Medical Research Council Cognitive Function and Ageing Study 2000). This finding may be a function of both the opportunity for direct observation and of a similarity in perspective attributable to similarity in age, education, and other factors (Qian and Preston 1993) that are known to influence evaluations of health care (Elliott et al. 2001; Zaslavsky et al. 2001).
In nursing homes and other institutional settings, spouses and family members are often less readily available than nonrelative adults, such as health care workers. Several studies examined the similarity of these proxy responses to those from spouses and other relatives (Rubenstein et al. 1984; Becchi et al. 2004; Kane et al. 2005; Kutner et al. 2006). The answer may depend on the subjectivity of the measure. For example, compared with relatives, nurse proxy reports on number of instrumental activities of daily living are closer to patient self-reports (Rubenstein et al. 1984), but nonrelative and relative proxies reports on nursing home resident quality of life were about equally close to resident self-reports.
When beneficiaries are unable to provide responses independently to the CAHPS® Medicare Fee-for-Service (MFFS) and Medicare Managed Care (MMC, now Medicare Advantage or MA) surveys, proxies are permitted to participate in ways that range from assistance (by reading the questions, writing down the answers the beneficiary gives, translating the questions into the beneficiary's language, or helping in some other similar manner) to serving as a proxy respondent (i.e., answering about the beneficiary's experiences in place of the beneficiary).
The current practice in the CAHPS MFFS is to use case-mix adjustment (CMA) with “assisted” and “proxy respondent” cases distinguished from “unassisted” cases, controlling for age, self-rated health, and education (Zaslavsky, Zaborski, and Cleary 2000; Elliott et al. 2001; Zaslavsky et al. 2001). This approach has found consistently less positive evaluations when proxy respondents were used (Zaslavsky, Zaborski, and Cleary 2000; Elliott et al. 2001; Zaslavsky et al. 2001), but the reasons for these differences have not been investigated.
When subgroups, such as unassisted and proxy respondent cases, are very dissimilar, CMA and other regression methods by themselves may produce biased estimates of the effect of proxy use if the regression model is misspecified. This may occur because proxy status is not typically randomly assigned and standard regression methods give equal weight to all cases, including those cases with almost no chance of membership in the group to which they are being compared.
To address this limitation, we employed propensity score weighting (PSW) to focus the comparison on the subgroup of beneficiaries who most resembled the beneficiaries who employed assistance or proxy respondents (Hirano and Imbens 2001). The propensity score is the probability that an individual belongs to a naturally occurring treatment group based on the individual's characteristics (Rosenbaum and Rubin 1983). This approach approximates inference under experimental assignment of treatment group under the assumption that there are no omitted variables in the propensity model relevant to selection into treatment. In practice, the technique may greatly reduce bias due to selection of the less healthy into the proxy treatment group and result in a more accurate measure of the effects of proxy use even when the assumption is not fully met, because it makes the regression model less sensitive to misspecification (Robins, Hernan, and Brumback 2000). Similarly, the use of covariates in regression in combination with PSW increases the robustness to misspecification of the propensity model (Robins et al. 2000). The PSW technique has been demonstrated to substantially reduce unadjusted differences in between self- and proxy reported health among Medicare beneficiaries (Ellis, Bannister, and Cox 2003). We hypothesize that PSW will also reduce the estimated size of proxy effects on evaluation of care received.
Our study investigates proxy effects so that we can better measure the care received by vulnerable recipients of health care. We explore the extent to which observed differences in ratings and reports of care are likely to be a result of actual patient experience and unobserved selection. We also assess whether rating differences vary as a function of characteristics of the survey items and by the nature of full proxy respondents. This study extends the existing literature by (1) estimating proxy effects among those who cannot or will not self-respond in dyad studies and (2) examining the extent to which mere assistance may influence survey responses. Additionally, we compare proxy estimates obtained from the standard regression-based approach with estimates obtained using PSW, bringing a new approach to the study of proxy effects on responses.
We analyzed data from the national 2001 CAHPS MFFS survey (3.0), which had a 68 percent response rate. Just under 5 percent of completed surveys (4.8 percent) were excluded from these analyses because information on the use of proxies was unavailable. Of the 110,215 remaining beneficiaries, 5 percent (n=5,361) had someone respond for them (“proxy respondent”), 14 percent (n=14,962) received help from a proxy (“proxy assistance”), and 82 percent (n=89,892) received no help (“no proxy”). The relationship of the proxy to the beneficiary was reported for 98 percent of proxy respondent cases and 94 percent of proxy assistance cases. Spouses and life partners (n=8,239) comprised 43 percent of proxies of either type; other family members (n=8,930) comprised 46 percent (two-thirds of these being children); and the remaining 11 percent (n=2,113) were not family members.
We analyze four global ratings (personal doctor, specialist, overall health care, Medicare) using an 11-point response scale, anchored only at 0 (“worst possible”) and 10 (“best possible”). We also analyze 23 report items that are grouped into five composites: getting needed care, getting care without long waits, how well doctors communicate, courtesy and helpfulness of office staff, and customer service. Report items used one of three response scales:
Global ratings are overall subjective assessments, whereas report items are more specific and thought to be more objective (Cleary et al. 1998). We disaggregate composites into report items in order to examine proxy effects at the item level.
Seven categorical variables were considered for our PSW and regression models: survey-reported age (18–64, 65–69, 70–74, 75–79, 80–84, 85+ years), education (eighth grade or less, some high school without completion, high-school graduate or general educational development diploma, 1–3 years of college, 4-year college graduate, more than 4-year college degree), race/ethnicity (Hispanic, non-Hispanic white, black, Asian, Asian Pacific Islander, Native American, mixed), self-rated general health (poor, fair, good, very good, excellent), self-rated mental health (same categories), SF-12 version 1 physical component and mental component scores, and proxy status (proxy respondent, proxy assistance, no proxy).
Two propensity score models were estimated as logistic regressions. Model A estimated the probability of proxy response or assistance, p(a), as opposed to no proxy (1−p(a)), among all respondents. Model B estimated the probability of proxy response p(b), as opposed to assistance (1−p(b)), among respondents with either form of proxy response. Weights of (1−p(a))/p(a), 1/(1−p(b)), and 1/p(b) for no proxy, assistance, and proxy respondents cases, respectively, can be shown to weight each group to resemble the set of respondents with proxies of either type (Robins et al. 2000; Hirano and Imbens 2001).
In building each of the two models, each candidate predictor was tested, one at a time, in a series of “bivariate” logistic regressions; significant terms (p<.05) were retained for a multivariate model. Ordinal variables were treated as categories if they departed significantly (p<.05) from linearity. Continuous predictors were tested for quadratic effects, which were retained if significant. Within each series, a main effects logistic regression was fit with all retained terms; all possible two-way interactions from this model were then added, with only significant interactions retained for the final pair of models.
We used the standard CAHPS CMAs (Zaslavsky, Zaborski, and Cleary 2000; Elliott et al. 2001; Zaslavsky et al. 2001) predicting ratings and reports from demographic characteristics, including level of proxy use (proxy respondent, proxy assistance, or none) and compared standard results with those derived by using propensity score weights in these same regressions.
In the first regression model (Model 1) we determined the effects of proxy status on beneficiaries' experiences with and without PSW. We predicted each of 27 outcomes from a proxy respondent dummy, a proxy assistance dummy, and dummies for categorical case-mix adjustors (age, education, race/ethnicity, self-rated general health, and self-rated physical health).
We also examined whether mean outcomes differed by the relationship of proxy respondents to the beneficiary using a modified regression model (Model 2). This model is the same as Model 1, except that the single dummy for proxy respondent is replaced by three dummies: spouse proxy respondent, other relative proxy respondents, and nonrelative proxy respondent. We examine Model 2 with and without the same PSW described for Model 1; these models exclude the few cases in which the relationship of proxy respondents is not reported.
Table 1 describes the demographics and health of respondents, overall and by proxy status. Compared with those not using a proxy, beneficiaries represented by proxy respondents were less educated, in worse physical health, older, and to a lesser extent, in worse mental health (p<.001 for all).
Beneficiaries who used proxy assistance are much less educated, more likely to be Hispanic or black, and have poorer physical health and slightly worse mental health than those not using proxies (p<.001 for all). Beneficiaries using proxy assistance were more likely than unassisted beneficiaries to be age 80 or older, but were also more likely to be under age 65 (which as a Medicare beneficiary indicates eligibility through disability), p<.001 for each. This latter finding is unsurprising, as several of the forms of proxy assistance (reading or writing for the beneficiary) reflect aid that may be necessitated by disability.
Table 2 shows the estimated effect sizes (nominal coefficients divided by the standard deviation of each outcome) for the four rating and 23 report items associated with use of a proxy respondent and with proxy assistance based on the PSW Model 1. Thus a value of “1” for proxy respondent would mean that answers from proxy respondents were more positive than those from self-respondents by one standard deviation of the outcome, after adjustments. As can be seen, proxy respondents are associated with consistently less favorable global ratings (average effect size −0.205; p<.05 for each) and differ less consistently on reports (significantly less favorable for eight and significantly more favorable for four of 23 report items, average effect size 0.065 in absolute value).
Proxy assistance is associated with less favorable global ratings (average effect size is −0.095; p<.05 for three of four), though the effect is only half the size observed with proxy respondents. Effects of proxy assistance on report items were small and mixed (five positive and five negative at p<.05, with an average absolute value of 0.039, about half that for proxy response).
Simple (unweighted) CMA results in estimated proxy respondent effects that differ from the 16 statistically significant PSW CMA estimates by as much as 55 percent (with a median difference of 22 percent), with CMA estimates generally larger than PSW CMA estimates (results not shown).
Table 3 summarizes the proxy respondent effect according to relationship to proxy (from Model 2). Spouse proxy respondents were more positive than other family members and nonfamily members; other family members and nonfamily members were virtually identical in their proxy respondent effects. Spouses were significantly more positive than other proxy respondents for all global ratings (mean effect size 0.26 standard deviations versus other family proxy respondents and 0.19 standard deviations versus nonfamily proxy respondents, p<.05 for all comparisons). Spouses were significantly more positive than other family members for 18 of 23 report items and significantly less positive for two (p<.05, mean effect size 0.10 standard deviations, largest effect size 0.32 standard deviations); spouses were significantly more positive than nonfamily proxy respondents for 16 of 23 report items and significantly less positive for one (p<.05, mean effect size 0.10 standard deviations, largest effect size 0.24 standard deviations, results not shown).
Spouse proxy respondents did not significantly differ from expected self-reports in ratings of physicians and specialists and were within 0.08 standard deviations for ratings of Medicare and health care received. Other family and nonfamily proxy respondents were generally similar to one another and provided global ratings that averaged 0.25 standard deviations lower than what would have been expected without proxies (p<.001 for seven of eight comparisons).
On subjective ratings, nonspouse proxy respondents rate care less positively than unassisted matched beneficiaries and perhaps less positively than these vulnerable beneficiaries would have themselves, though the effects of unobserved differences cannot be ruled out. For report items designed to be more objective and specific, proxy differences are smaller and less consistent.
These findings are consistent with prior research that concludes that proxy respondents can both over- and underestimate morbidity and disability (Magaziner et al. 1988; Sneeuw et al. 1997; Shaw et al. 2000; Todorov and Kirchner 2000; Duncan et al. 2002; Tang and McCorkle 2002; Pickard et al. 2004). By extending this work through new methodology to populations that are less likley to self-respond, we strengthen the evidence for the generality of previous findings.
This study demonstrates how the PSW approach can be used to confirm, modify, or extend earlier health services research by accounting for selection into treatment groups on the basis of observable characteristics and making regression approaches less sensitive to misspecification. Here we apply this technique to the study of proxy effects on responses to health care experiences surveys. The dyad methodology characterizing prior research in this area is applicable only when recipients of health care are willing and able to self-respond; PSW allows us to extend the scope of this research into the important settings in which individuals who would not have self-responded use proxies. The importance of PSW in this observational setting is highlighted by findings that standard CMA may overestimate proxy effects in this context by failing to fully account for differences in observable characteristics. If this Medicare population represents those typically requiring proxies in the evaluation of their health care, PSW may be a useful technique to eliminate or reduce selection bias and to obtain more accurate measures of the effects of proxy use on evaluations of health care in nonexperimental settings where beneficiaries self-select in the use of proxies.
Health care evaluations and other survey responses may be influenced by respondent characteristics, even when underlying experiences do not differ (Elliott et al. 2001; Zaslavsky et al. 2001). There are at least four reasons that the responses of beneficiaries using proxies might differ from those of unassisted beneficiaries: (1) beneficiary differences in observed characteristics related to survey response or difficulty to treat (e.g., poorer health, lower education), (2) beneficiary differences in unobserved characteristics (e.g., specific cognitive impairments limiting communication), (3) proxies might report differently on the same experience than an unassisted beneficiary would (or might influence reports through assistance), and (4) proxies might report beneficiary characteristics used for CMA differently than the beneficiaries would.
In this study, we interpret estimates that use both covariates and PSW as nonexperimental estimates of the effect of proxy respondents or proxy assistance on evaluations of care (corresponding to explanation 3), rather than mere descriptions of differences between those using and not using proxy respondents or proxy assistance, given a careful attempt to control for explanation 1 (the effects of observable characteristics). Because beneficiaries are not randomly assigned to proxy use, some caution is warranted in inferring the causality of proxy effects when comparing matched beneficiaries. Our study design is such that we cannot unambiguously distinguish the remaining differences between proxy and self-respondents from those attributable to unobserved heterogeneity or from differences in how beneficiaries and proxies report on the same experiences or characteristics. For example, if selection into proxy use is imperfectly modeled, we cannot rule out explanation 2.
While the present study cannot directly assess the existence of proxy effects on case-mix variables (explanation 4), the indirect nature of this effect and the relatively moderate associations between CAHPS case-mix adjustors and outcomes (Zaslavsky Zaborski, and Cleary 2000; Elliott et al. 2001; Zaslavsky et al. 2001) means that this factor is unlikely to be important unless proxy effects on demographics are very large compared with proxy effects on evaluations of care. Spouse/nonspouse findings also provide evidence against explanation 4. Previous research (The Medical Research Council Cognitive Function and Ageing Study 2000) suggests that proxies who live with a respondent (who are more often similarly aged spouse proxy respondents) are likely to rate the health status of beneficiaries more positively than the generally younger other family members (mostly children). Given that CMA adjusts downward for better health, proxy effects on this one important case-mix adjustor would run contrary to observed differences between spouses and family members as proxies.
We also find that proxies who merely “assist” the beneficiary may influence response, though less than if they answer for the beneficiary. Further research, such as cognitive interviews or observation of proxy assistance in action, might investigate mechanisms by which proxy assistants can influence global ratings. For example, assistance may involve some interpretation on the part of the proxy, rather than literally conveying the context of the survey or of the beneficiary response.
If a survey anticipates a high proportion of respondents who will need to employ a proxy respondent or assistance (e.g., a nursing home or home health care setting), a conservative approach would be to place greater reliance on specific and objective reports than on subjective ratings (supposing that unmeasured proxy effects may follow a pattern similar to what is seen with observed proxy effects). When available, spouses may be especially desirable as proxies, given their closer correspondence to what would have been expected from self-report. When nonspouse proxy respondents are used and they make up a substantial proportion of responses, adjustments should consider the relationship of proxies to respondents.
The authors would like to thank Kate Sommers-Dawes for assistance with the preparation of the manuscript. We would also like to thank our project officers, Ted Sekscenski and Amy Heller. Support for preparation of this manuscript for this research comes from the Center for Medicare and Medicaid Services (CMS) through a subcontract to RTI International, which subcontracts from the University of Wisconsin (contract number 2-5IU-7903 and from a CMS contract to RAND HHSM-500-2005-00281). Marc Elliott is supported in part by the Centers for Disease Control and Prevention (CDC U48/DP000056). Ron Hays is supported in part by the National Institute on Aging (NIA) (AG20679-01).
The contents of the publication are solely the responsibility of the authors and do not necessarily represent the official views of CMS, the CDC, or the NIA.