|Home | About | Journals | Submit | Contact Us | Français|
To assess the effect of case-mix adjustment on community health center (CHC) performance on patient experience measures.
A Medicaid-managed care plan in Washington State collected patient survey data from 33 CHCs over three fiscal quarters during 2007–2008. The survey included three composite patient experience measures (6-month reports) and two overall ratings of care. The analytic sample includes 2,247 adult patients and 2,859 adults reporting for child patients.
We compared the relative importance of patient case-mix adjusters by calculating each adjuster's predictive power and variability across CHCs. We then evaluated the impact of case-mix adjustment on the relative ranking of CHCs.
Important case-mix adjusters included adult self-reported health status or parent-reported child health status, adult age, and educational attainment. The effects of case-mix adjustment on patient reports and ratings were different in the adult and child samples. Adjusting for race/ethnicity and language had a greater impact on parent reports than adult reports, but it impacted ratings similarly across the samples. The impact of adjustment on composites and ratings was modest, but it affected the relative ranking of CHCs.
To ensure equitable comparison of CHC performance on patient experience measures, reports and ratings should be adjusted for adult self-reported health status or parent-reported child health status, adult age, education, race/ethnicity, and survey language. Because of the differential impact of case-mix adjusters for child and adult surveys, initiatives should consider measuring and reporting adult and child scores separately.
Community health centers (CHCs) have long been recognized as important safety net providers of primary care. They now serve over 16 million patients every year, many of whom are ethnic and racial minorities, low income, and uninsured or publicly insured (National Association of Community Health Centers 2007). Recent studies have found that CHCs provide health care of comparable quality to private physician practices and hospital outpatient departments (Hedberg et al. 1996; Ulmer et al. 2000; Frick and Regan 2001; O'Malley and Mandelblatt 2003; Regan et al. 2003; Shi et al. 2003;). However, evidence suggests that the quality of care varies significantly across CHCs and not all health centers meet national benchmark standards of care (Ulmer et al. 2000).
Patient experience measures, such as the Clinician & Group Consumer Assessment of Healthcare Providers and Systems (C-G CAHPS) survey (Agency for Healthcare Research Quality et al. 2006), are widely used to evaluate and improve quality of health care from patients' perspectives. Patients' experiences of care have been associated with adherence to treatment recommendations (Safran et al. 1998; Schneider et al. 2004;), positive health outcomes (Stewart 1984; Stewart et al. 2000;), and loyalty to physicians (Safran et al. 2001; Rodriguez et al. 2007;). Patients are also increasingly using patient-reported information about individual physicians and care sites to inform their selections of primary care arrangements (Fung et al. 2005; Fanjiang et al. 2007;).
As managed care organizations become more involved in the provision of health services for Medicaid patients (Centers for Medicare and Medicaid Services 2009), CHCs are increasingly required to report summaries of patient experience measures for quality improvement purposes and some CHCs are subject to pay-for-performance incentives (Coleman, Reiter, and Fulwiler 2007; Felt-Lisk, Gimm, and Peterson 2007;). Many quality improvement initiatives struggle with equitably comparing clinical quality and patient experience performance across CHCs because they serve high proportions of disadvantaged patients and face great difficulties recruiting and retaining physicians, administrators, and other personnel (Singer et al. 1998; Baer, Konrad, and Miller 1999; Cochran and Peltier 2003; Rosenblatt et al. 2006;). Health care providers who serve high concentrations of vulnerable patients report significantly more resource and organizational constraints that may limit their ability to provide high-quality care (Varkey et al. 2009).
Research has consistently indicated that certain patient characteristics, such as age and health status, are associated with ratings of care and may reflect patient reporting tendencies rather than true differences in the quality of care received (Cleary and McNeil 1988; Hall et al. 1990, 1998; Zapka et al. 1995). For Asian or Latino patients, lower reports do not always coincide with lower overall ratings of care received (Weech-Maldonado et al. 2001, 2003). These discrepancies may stem from the greater influence of expectations or reporting tendencies on ratings, while reports of specific health care experiences may be more objective (Morales et al. 2001; Weech-Maldonado et al. 2001, 2008). Such findings highlight the importance of assessing patient reports of specific health care experiences and overall ratings of care separately. Adjusting patient experience measures for patient characteristics that are not within the control of the units being evaluated, or “case-mix adjustment,” may improve the reliability and perceived validity of patient-reported experiences of care measures.
Although case-mix adjustment has been extensively researched at the health plan level (Zaslavsky et al. 2001; O'Malley et al. 2005; Eselius et al. 2008;), little is known about the effects of case-mix adjustment on summaries of patient experience measures for CHCs. The extent to which the effect of case-mix adjustment differs between adult-reported experiences of care and parent-reported experiences of pediatric care also remains understudied and is an important consideration for CHCs, which generally serve patients of all ages. One study that assessed differences between adult and pediatric surveys found that, among commercially insured patients, adult/parent respondent characteristics had similar adjustment effects on reports (Zhan et al. 2002). However, these differences have not been examined for CHCs, where patient populations are more often socioeconomically disadvantaged and racial/ethnic minorities.
The primary objective of this study was to assess the effect of case-mix adjustment on CHC performance on patient experience measures in an effort to more equitably compare CHC performance. The results could provide valuable information for how widely used patient experience measures, such as the C-G CAHPS surveys, are influenced by patient characteristics at different primary care delivery sites. Based on previously reported discrepancies between patient reports and ratings of primary care experiences, we chose to analyze the effects of case-mix adjustment for these two types of measures separately. In addition, we assessed differences in the impact of adjustment on patient experience measures between adult and pediatric surveys in order to inform the extent to which mixes of child and adult survey responses can affect the performance of CHCs.
This study uses patient-reported survey data from a large Medicaid managed care plan in the state of Washington from three fiscal quarters. Data were collected in three waves, beginning in the last quarter of 2007 and continuing through the first two quarters of 2008. Patients were eligible to respond if they had seen a health care professional at a CHC in the managed care network during the prior 6 months and had continuous coverage over the previous year. These responses came from 33 CHCs (23 federally qualified health centers and 10 other safety-net clinics). Respondents were contacted by phone and interviewed in their language of preference. Patients who did not have a working phone were excluded from the sample.
Patient interviews elicited information about health care experiences for their (or their child's) visit at a named CHC in the previous 6 months. Adult respondents answered questions about their own or their child's health care experiences during this visit. A total of 5,106 adults were interviewed, resulting in a 32 percent response rate. Among the respondents, 2,247 were adults responding about their own care and 2,859 were responding about their child's care. There was an average response of 86 adult surveys and 86 pediatric surveys for each CHC. Most interviews were conducted in English (68.7 percent), followed by Spanish (26.3 percent), Chinese (1.7 percent), Vietnamese (1.7 percent), and Russian (1.6 percent).
The survey used measures of patient health care experiences adapted from the Clinician & Group CAHPS instrument (see Appendix SA2). The patient experience reports consisted of 12 items capturing three domains of care (physician communication [4 items], access to care [6 items], and office staff interactions [2 items]). These individual report items were measured on a four-point (“Never,”“Sometimes,”“Usually,”“Always”) or five-point (“Very Dissatisfied,”“Somewhat Dissatisfied,”“Neither,”“Somewhat Satisfied,”“Very Satisfied”) response scale. Composites were computed based on the unweighted average of all items in each domain of care (α range: 0.81–0.82). Two global ratings of health care asked patients to rate their overall satisfaction with their health care and physician. These ratings were reported on a 0–10 scale where 0 represented “the worst possible care” and 10 represented “the best possible care.” All scores were converted to a 100-point scale to facilitate the comparison of case-mix effects across domains of care.
Adult respondent demographics included age, gender, race/ethnicity, educational attainment, self-reported health status, and language. Adults were considered to be Latino if they reported that they were of Hispanic ethnicity, irrespective of their racial identification. For pediatric surveys, parents or caregivers reported the children's health status and age.
In our preliminary analysis to select potential case-mix adjusters, 12 patient reports and 2 global ratings of care were used as dependent variables for both the adult and child samples. The following seven respondent characteristics were tested as potential case-mix adjusters: self-reported health of adult or parent-reported health of child, respondent age, child age, respondent gender, respondent education, respondent race/ethnicity, and non-English survey language. To assess differences in the distribution of respondent characteristics across the two samples, χ2 tests were used for dichotomous variables and Wilcoxon tests were used for ordinal variables. Because there were several statistically significant differences between respondent characteristics across the adult and child survey samples (e.g., adult age, gender, and education), we stratified the analyses by survey type.
Our criteria for selecting case-mix adjusters was explanatory power (Zaslavsky et al. 2001), which identifies case-mix adjusters that are both significantly associated with patient-reported experience measures and vary across the unit of analysis, in our case, CHCs. As Eselius et al. (2008) note, a patient characteristic that is significantly associated with quality scores but does not vary across the unit of analysis would not be an appropriate case-mix adjuster. For example, respondent age might be highly correlated with patient assessments of their health care experiences, but if the distribution were the same across all CHCs it would not be necessary to adjust for age. We assessed potential case-mix adjustors for reports and ratings separately, based on evidence from previous studies suggesting that there are differences in reporting tendencies for these two types of measures (Weech-Maldonado et al. 2001, 2003).
The explanatory power for each potential case-mix adjuster was calculated by multiplying predictive power by the variance ratio. To calculate predictive power, null linear regression models for each patient experience item (12 reports and two ratings) were specified, controlling for the clustering of patients within CHCs using CHC fixed effects. Potential case-mix adjusters were individually added to the null models and the differences in R2 were calculated (predictive power). A variance ratio of between CHC variance to within CHC variance was calculated for each adjuster and multiplied by 1,000, for legibility.
The explanatory power calculation allowed us to compare the relative importance of potential case-mix adjusters. In the preliminary analysis, adult self-reported health status in the adult sample and parent-reported child health status in the child sample had the highest average explanatory powers and largest number of significant associations with patient experience measures and were included in our base model. All other variables were evaluated for their incremental explanatory power after controlling for health status. Variables that had both a relatively high explanatory power and predictive power were included in the final case-mix models.
Multivariate regression models for each of three composite measures and two global ratings accounted for CHC fixed effects and included the selected case-mix adjusters as independent variables. We calculated both the adjusted and unadjusted CHC mean scores for the global ratings and composite measures. In order to compare alternative case-mix models, adjusted and unadjusted scores were centered at zero and we calculated the largest positive, largest negative, and largest absolute change in scores. Finally, a Kendall's τ correlation coefficient was used as a summary measure to evaluate the impact of adjustment because it gives the best overall picture of how adjustment affects CHC rankings. A Kendall's τ of 1 would indicate that CHCs were ranked in exactly the same order, both before and after adjustment. This correlation coefficient also allowed us to quantify the percent of all possible CHC pairs that changed order after the adjustment. We interpreted a Kendall's τ correlation coefficient of 0.8 to mean that 10 percent ((1–0.8)/2) of all possible CHC pairings changed order after adjustment. In our sample of 33 CHCs there are 528 possible ways to order pairs and if all pairs switch order, the adjusted rankings would be in the exact opposite order of the unadjusted rankings.
Table 1 presents descriptive statistics of the two study samples. We found significant differences between the respondent characteristics of the adult and child survey samples. Adults responding to the child survey were more likely to be female (82.3 percent versus 76.4 percent, p<.001), younger (median age group 23–34 versus 35–44, p<.001), nonwhite (59.5 percent versus 46.0 percent, p<.001), and non-English speaking (36.2 percent versus 25.2 percent, p<.001). Finally, respondents to the adult survey had a slightly higher proportion of reporting “some college” and “college graduate.”
A total of 14 regressions (12 reports and 2 global ratings) were estimated for each potential case-mix adjuster. Regressions evaluating associations of reports or ratings with health status controlled only for fixed effects. All other potential case-mix adjusters were added individually to linear models controlling for adult self-reported health status or parent-reported child health status and CHC fixed effects. Table 2 shows the total number of these regressions in which patient characteristics had significant associations with reports or ratings. Of note, adult age was positive and significant in 5 out of 12 models of patient report items in the adult sample and 6 out of 12 models in the child sample. However, adult age was not significantly associated with global ratings in either sample.
The associations of patient characteristics with overall ratings were largely consistent in direction across both samples. Education coefficients (higher education) were uniformly negatively and significantly associated with global ratings. Latino adults and non-English speakers provided significantly higher overall ratings of care compared with other racial/ethnic groups and English speakers. However, there are some notable differences between the adult and child samples in the associations of patient characteristics with report items. In the adult sample, few significant associations with reports were observed for racial and ethnic groups, while in the child sample Latino parents reported significantly more negative experiences of care for 6 out of 12 report items. Although non-English survey language was inconsistently associated with reports in the adult sample, it was negatively associated with 5 out of 12 report items in the child sample.
Table 2 shows the average explanatory power for each potential adjuster, averaged separately over 12 report items and two global ratings of care for the adult and child samples. Average explanatory power for reports was higher in the child sample than the adult sample for most race/ethnicity variables and non-English survey language. Explanatory power of Latino ethnicity for reports was 2.51 in the child sample compared with 0.73 in the adult sample.
The explanatory power of patient characteristics, including education, language, and Latino ethnicity, was generally higher for global ratings than reports. For ratings, Latino ethnicity had the highest explanatory power in both samples. Education and non-English survey language also had high explanatory power in both samples, though the values in the adult sample were nearly double those observed in the child sample (6.27 compared with 2.08, and 6.30 compared with 3.45). Although adult age did not have high explanatory power for reports or ratings in either sample, it was included in adjustment models because it has been consistently shown to be an important case-mix adjuster in diverse patient samples (Kim, Zaslavsky, and Cleary 2005; Eselius et al. 2008;), and because it had a large number of significant associations with patient experience measures in both study samples.
The final models for adults and children included adult self-reported health status or child health status, adult education, and adult age (Table 3, Model 1). We also compared the incremental effect on composite scores and ratings when adult race/ethnicity and language were included in the model (Table 3, Model 2). The largest downward change (6.27 points) in summary score for an individual CHC was for overall rating of medical care in the adult sample. Meanwhile, the largest upward change (7.04 points) was for access to care in the child sample. The mean absolute adjustments reported in Table 3 describe the average change in CHCs scores after adjustment, without regard to the direction of the effect.
For both the adult and child samples, the impact of adjustment was greater when race/ethnicity and language were included in models. However, this impact was even larger for the child sample. The mean absolute adjustment for the access composite increased from 0.74 to 1.38 in the child sample when race/ethnicity and language were added to the model, while it increased from 0.62 to 0.77 in the adult sample. This differential adjustment effect remained consistent across all composite measures except for office staff interactions, where both samples experienced similar adjustment effects. These results underscore that combining composites scores from adult and child surveys to assess performance of CHCs may be inappropriate because case-mix adjusters have different effects across samples.
The adjustment's impact on relative rankings of CHCs is shown in Table 4. The impact of adjustment on relative rankings was greater for Model 2 than for Model 1 in both samples. For overall ratings of medical care, the percent of all possible CHC pairs to change order increased from 10.5 percent to 14.9 percent in the adult sample and from 5.0 percent to 10.7 percent in the child sample. However, the overall effect of adjustment was modest for most composites and ratings, with a Kendall's τ of 0.8 or higher in most cases. There were some notable exceptions, such as the adjustment effect observed for Model 2 on global ratings of care in the adult sample (Kendall's τ=0.70) and on the physician communication composite in the child sample (Kendall's τ=0.72). Although the effects of adjustments on composites and ratings were generally modest, they did impact the relative rankings of some CHCs.
This study is the first to assess the effect of case-mix adjustment on CHC performance on patient experience measures. We found that statistical adjustments have important, though modest, effects on CHC performance and alter the relative ranking of CHCs. If pay-for-performance is implemented as a quality improvement strategy, differences in rankings would affect how financial incentives are distributed and could have unintended negative consequences for CHCs serving high proportions of ethnic and racial minority patients (Casalino and Elster 2007). In particular, CHCs under financial strain (Shi et al. 2001; McAlearney 2002; Cunningham, Bazzoli, and Katz 2008;) that perform worse on patient experience measures might avoid seeing patients that are thought to contribute to lower scores. Conversely, CHCs that perform well on experience measures that are not adjusted accurately may not recognize care processes that need improvement. CHC clinicians and leaders might also be skeptical of patient-reported health care experience information if differences in patient mix are not considered in scoring methods.
Consistent with previous research, our results indicate that adult self-reported health status and adult education are important case-mix adjusters (Zaslavsky et al. 2001; Kim, Zaslavsky, and Cleary 2005; O'Malley et al. 2005; Eselius et al. 2008;). Although we included adult age in our models for face validity, we did not find the same explanatory power that has been reported in previous studies (Kim, Zaslavsky, and Cleary 2005; Eselius et al. 2008;), due to the relatively low predictive power of adult age across both samples. In our population, self-reported health status mediated much of the effect of age on patient-reported experiences.
Our findings differ from previous research among commercially insured patient samples, which suggested that adult respondent characteristics have influences on patient experience measures similar in size and direction for both adult and child surveys (Zhan et al. 2002). We observed differences between adult and child surveys in the case-mix effects of adjusters, including differences in their explanatory powers and coefficients. In particular, the explanatory powers of health status, Latino ethnicity, and survey language were generally larger in the child sample for report items. If the case-mix effects are not uniform across samples, the coefficients reflect differences in how CHCs serve either adult or child patients (Zaslavsky, Zaborski, and Cleary 2000). Combining samples would require extensive statistical adjustment for differential effects using interaction terms and scores would still inadequately adjust performance for case-mix effects.
We also found that including race/ethnicity and non-English survey language in our case-mix model (Model 2) had a greater impact on patient experience summaries and rankings than that observed for Model 1. This case-mix effect was even more pronounced in the child sample, where we observed larger explanatory power of race/ethnicity and non-English survey language for reports than in the adult sample. The greater between-CHC variability of race/ethnicity and language was the primary driver of these results, as minority and non-English proficient parents reporting on their child's health care experiences were more clustered at CHCs than other adults. The effects of race/ethnicity or language may also be stronger in the child sample because parents have been found to be less tolerant of perceived problems with their child's primary care experiences and more readily seek alternatives if they perceive quality to be better elsewhere (Berry et al. 2008).
Our results are consistent with previous research findings of differences between patient-reported health care experiences and overall ratings (Morales et al. 2001; Weech-Maldonado et al. 2001, 2003). Low English proficiency has been shown to have a negative association with patients' experiences of physician interactions (Garcia et al. 2004) and positively associated with patients' global ratings of care (O'Malley et al. 2005). Similarly, we found that non-English survey language was negatively associated with patient reports on access and physician–patient interactions, but a positive effect on overall ratings. There are many possible explanations for these lower patient-reported experiences. Physicians may not be communicating effectively with low English proficient patients, with our without professionally trained interpreters, and language barriers may also impede timely access to care at some CHCs. The discrepancy between negative reports and positive ratings may stem from the greater influence of expectations on ratings (Morales et al. 2001; Weech-Maldonado et al. 2001;), or from extreme reporting tendencies, in which respondents are more likely to select extremely high or extremely low scores (Weech-Maldonado et al. 2008). More research is needed to determine the nature of the relationship between language proficiency and reporting tendencies.
There is some debate about whether to include race/ethnicity variables in case-mix models. It has been proposed that adjusting for race/ethnicity may actually discourage health care organizations from implementing systems to remove health care disparities (Romano 2000). Yet different racial and ethnic groups have been consistently shown to report their experiences of health care differently. For example, Asian patients tend to give more negative reports of their health care experiences (Saha and Hickam 2003; Rodriguez et al. 2008;). Adjusting for all patient demographics that affect scores ensures that summaries of composite scores for CHCs reflect the scores that CHCs would have been given if they were all serving the same standard population and do not merely represent differences in the populations served (Zaslavsky, Zaborski, and Cleary 2000). Although a preferable practice may be to report performance measures separately by race/ethnicity (Fiscella et al. 2000), achieving adequate sample sizes from all major racial and ethnic groups could be very costly. Moreover, racial and ethnic minority groups are significantly clustered in primary care practices with low performance on patient experience measures (Rodriguez et al. 2008), making stratifying results by race/ethnicity impractical.
Our study results should be considered in light of several important limitations. First, the survey had a low response rate of 32 percent and we do not have information about nonresponders. However, our response rate is similar to that achieved in other similar studies of patient experiences of care (Jha et al. 2008). Although response rates may be related to patient experiences of care, empirical evidence suggests that adjusting for nonresponse does not improve the precision of performance comparisons (Elliott et al. 2005). Furthermore, the relationship between response rates and scores on patient experience measures is often a product of differences in case-mix, and case-mix adjustment has been shown to eliminate much of the impact of nonresponse bias (Elliott et al. 2005; Campbell et al. 2009;).
Second, our sample was from the State of Washington and our results may not generalize to geographic areas with different demographic distributions. For example, in Washington State, a larger proportion of the state's Latino population is foreign born (41 percent), younger than 18 (38 percent), and of Mexican origin (80 percent) than in other states (Pew Hispanic Center 2009). Other states serving a different Latino population may observe different patterns of patient experience reporting, since evidence suggests that Mexican Americans are less integrated into primary care (Burnette and Mui 1999). Nevertheless, our findings highlight the importance of case-mix adjustment across CHCs and the methods could help to guide these efforts in other regions. Third, the data were obtained via telephone interviews, which are known to result in lower response rates and higher overall scores than other survey modes (Elliott et al. 2009). However, the use of telephone interviews also increases the representativeness of respondents because patients with low literacy levels are less likely to complete mailed surveys (Fowler et al. 2002). This is a particular concern for CHCs because most primarily serve patients of low socio-economic status.
Finally, recent studies have found that language has a greater effect on patient experiences than race/ethnicity (Weech-Maldonado et al. 2001, 2003). Although we suspect that the effect of race/ethnicity on patient reports and ratings was modified by language, we did not have a large enough sample size per clinic to stratify major racial/ethnic groups by survey language. Yet this is a limitation that many health care organizations face because of small samples for subgroups. Our findings suggest that both English proficiency and race have strong effects on patient experiences and should be included in case-mix adjustments. A recent report from the Institute of Medicine has highlighted the importance of accurately collecting information on patient race, ethnicity, and language preference in order to identify and address disparities in health care quality (Institute of Medicine 2009). As our findings demonstrate, standardized collection of patient demographics is also critical to accurately comparing CHC performance on patient experience measures.
In conclusion, our findings provide valuable information for managed care organizations that seek to improve patient experiences of primary care at CHCs. When comparing the performance on patient experience measures across CHCs, we recommend adjusting for the case-mix effects of adult self-reported health status or child health status, adult education, age, educational attainment, race/ethnicity, and non-English language. We also recommend analyzing adult and child reports and ratings of health care experiences separately. It is ideal to focus improvement efforts on primary care sites and individual clinicians because they account for most of the explainable variation on patient experience measures (Rodriguez et al. 2009). CHCs serving diverse patient populations often take pride in delivering culturally competent medical care. Strategies such as providing interpreter services, cultural competency training, recruiting and retaining diverse staff, using community health workers, and culturally competent health promotion have been proposed as ways to improve quality of care and reduce racial and ethnic disparities (Brach and Fraser 2000; Beach et al. 2005; Morales et al. 2006;). However, little empirical research has been conducted to examine how organizational systems and policies aimed at improving cultural competency impact performance on patient experience measures, such as physician–patient communication quality (Paez et al. 2008). Focused research on improving patient-centeredness in safety-net clinics and CHCs will be important for reducing racial and ethnic disparities in health care quality.
Joint Acknowledgment/Disclosure Statement: No funding was obtained to conduct the study analyses. We would like to thank the CHC patient respondents for their participation in the interviews. We are also grateful for the data collection and cleaning efforts made by staff at the Community Health Plan of Washington. The authors have no conflicts of interests to report.
Additional supporting information may be found in the online version of this article:
Appendix SA1: Author Matrix.
Appendix SA2: Patient Experience Measures.
Please note: Wiley-Blackwell is not responsible for the content or functionality of any supporting materials supplied by the authors. Any queries (other than missing material) should be directed to the corresponding author for the article.