Measures of medical care processes, including patient reports about their care experiences, are now widely used by consumers, purchasers, and accreditation organizations (NCQA 2005
). When comparing plans using patient reports of care experiences, it is important to adjust for patient characteristics that affect scores but are unrelated to differences in quality of care. In this study, we use data from more than 4,000 patients who had received care in 21 behavioral health care plans to develop a statistical model for adjusting scores for patient characteristics and examine the effects of adjustments using this model on plan scores and relative rankings.
Consistent with other CAHPS studies (Zaslavsky et al. 2001b
; Kim, Zaslavsky, and Cleary 2005
; O'Malley et al. 2005
), the average impact of case-mix adjustment on plan scores for ratings and reports collected from behavioral health care patients was modest. Adjustments did change plan rankings in a few cases for both the commercial and Medicaid plans, with adjustments typically being larger for the Medicaid plans. For a few individual plans, the change in some summary scores was large.
Although the effects in these data are modest, adjustments would be larger in groups of plans with greater interplan heterogeneity in patient characteristics. Whether the impact is large or small, case-mix adjustment may still be important to maintain the credibility of patient reports as a quality metric. In the absence of case-mix adjustment, plans that believe their patients have worse health status than patients in other plans may believe summary scores are suspect and be reluctant to rely on them as a quality indicator. The fact that case-mix adjustment can have a meaningful effect on plan scores and rankings, and requires only a small amount of information that is typically collected for other purposes such as subgroup analyses, makes it worthwhile to carry out. Doing so can preserve the face validity of the results for plans who might otherwise argue that their patients are more severely ill than is typical.
This study had several limitations. The sample consisted of plans that participated in the field test of the ECHO survey or voluntarily submitted their survey data to the National CAHPS Benchmarking Database. We do not know how well the findings of this study generalize to other plans. We suspect, however, that the main difference between participating and nonparticipating plans would be in the average scores, rather than in different patterns of associations. Response rates varied considerably. Limited analyses comparing respondents and nonrespondents indicated that younger individuals and men may be underrepresented in the sample, patterns that have been found in other behavioral health studies (Rosenheck, Wilson, and Meterko 1997
Although prospective studies also have shown that health status is associated with care experiences, a potential problem with case-mix adjustment based on self-reported health status is that the quality of behavioral health care received may lead to changes in health status. By controlling for health status when reporting scores, plans may fail to benefit from improving the health of their enrollees. While this effect may be real, the average change in health status due to differences in quality of care is likely to be small, relative to the underlying health status of the individual.
As expected, the self-reported health status measures were the strongest and most consistent predictors of ratings and reports among the personal characteristics included in this study. Mental health status was frequently a strong predictor, as in other studies (Zaslavsky et al. 2001b
), although general health status remained important in several cases after controlling for mental health status. This association may be due to general reporting tendencies that are associated, for instance, with general life satisfaction (Rohland, Langbehn, and Rohrer 2000
) or with effects of mental illness on mood and perception. Patients in worse mental health may also receive lower quality of care than patients in better mental health. Providers are likely to have more difficulty communicating with patients who are distressed and may tend to unconsciously convey negative attitudes and behaviors toward patients in poor mental health (Hall, Milburn, and Epstein 1993
). The chronic and recurring nature of many behavioral health conditions, and the uncertainty involved in determining the best treatment strategy for a particular patient, increase the likelihood that multiple treatment approaches will be attempted before symptoms are relieved, which may cause dissatisfaction with care.
Differences in reported care experiences between patients who received treatment for alcohol or drug use and other patients in the sample may reflect different treatment patterns for these two groups of conditions. For instance, those being treated for alcohol/drug use were more likely to report that providers had discussed different kinds of counseling or treatment and whether to include family and friends. In contrast, they gave less favorable reports about whether providers listened carefully, explained things, and showed respect for what they had to say. Furthermore, these respondents were less likely to report feeling confident that information about them was kept private by providers and more likely to report experiencing problems with delays in treatment while waiting for plan approval. Given the differences in these scores, and the types of treatments received, users of patient reports of care might consider separate ratings for individuals who report treatment for alcohol or drug use. In the current study, this group was too small to be analyzed separately.
As in other studies, older respondents tended to give more positive ratings and reports than younger respondents (Zaslavsky et al. 2001b
). Older respondents may have lower expectations regarding their care and/or more respect for providers, leading them to give more positive reports and ratings. Older patients also may receive better care than younger patients if, for instance, providers tend to be more attentive to older patients (Cleary et al. 1992
). Education was negatively associated with positive ratings. Because it is unlikely that respondents with more education receive worse care than respondents with less education, the findings are consistent with the hypothesis that highly educated respondents have higher expectations regarding their care, which result in less favorable assessments. The mixed findings related to income are consistent with other studies (Dow, Boaz, and Thornton 2001
; Heflinger et al. 2004
). Although a substantial number of respondents are reluctant to provide income data, an income item with coarse categories may be useful for case-mix adjustment when the income distributions of plan populations vary greatly. Because the effects of race/ethnicity were not the same in commercial and Medicaid plans, it may be important to estimate separate models for the two types of plans (as we do here) and make comparisons only among plans of the same type.
The regression models used to control for case-mix differences assume that the effects of adjusters are equal within each plan. While the effects of most demographic adjusters were similar across plans, the effects of mental health status varied across plans for several rating and report questions. Therefore, the adjusted plan scores, and possibly the rankings of plans based on those scores, would depend on whether the plans were compared with respect to how patients with poor, average, or excellent mental health status would rate them (Zaslavsky 1998
). Reporting separate summaries for patients in relatively poor and good health (provided that sample sizes for these groups were sufficient within each plan) may be important for identifying performance differences among plans.
In summary, mental health status, general health status, alcohol/drug use, age, education, and race/ethnicity were identified as relevant case-mix adjusters for the ECHO survey, although the case-mix adjustment model resulted in only modest changes to plan ratings and rankings.