The results of this study highlight the complexity that often underlies apparently simple, average measures used to monitor performance trends across organizations or geographic areas. In this case, we looked at performance on CAHPS patient experience-of-care measures in the Medicare program, for which the geographic areas being compared were U.S. states. In 2008, California ranked near the bottom among states in terms of physician services and immunizations, had mixed rankings for Part D measures, and had average-to-above average performance in plan services. These differences were generally small (an exception perhaps being a 4-percentage point gap in pneumococcal immunization), but they were practically and statistically significant. These results have informed our research questions regarding possible influence of different factors.
1. To what extent do overall differences in CAHPS performance between California and the rest of the country hold within the fee-for-service and MA sectors?
We found strong differences between the Medicare MA and fee-for-service sectors. In domains of immunization, plan services, and Part D services, California MA consistently exceeded the national MA average, whereas California fee-for-service generally lagged the national average. Both sectors were below average for physician services, but more so for California MA. These results are consistent with findings from other studies that found variations in performance across different quality measures (Miller and Luft 2002
; Landon et al. 2004
; Gillies et al. 2006
These differences were not unexpected because the two health care models are quite different. Beneficiaries in fee-for-service Medicare have standard Medicare benefits and can choose their physicians and other providers freely. Beneficiaries enrolled in the MA plans have expanded benefits and the plans more actively manage their care processes and access to providers, with some plans essentially locking the beneficiary into receiving care from a single group practice. Taken as a whole, California MA plans may exhibit the traditional management characteristics of managed care more strongly than other Medicare managed care nationally.
2. Are there additional stable characteristics of the beneficiaries or market that would explain observed score differences between California and the rest of the country, but which are not included in the standard CAHPS case mix adjustments?
Given the large differences between California and the rest of the country in race/ethnicity and urbanicity, we anticipated that their addition as case mix adjustors might be sufficient to explain the differences. While the additional adjustors improved California scores on almost all measures, they did not fully explain the lower California scores in MA and they explained fewer than half the lower California scores in fee-for-service. The most important contributor was race/ethnicity; urbanicity had much weaker effects, making the question of whether urbanicity captured response tendency or quality of care moot for this application. The higher prevalence of Asians (who tend to provide the lowest ratings and reports of care) and Hispanics (who tend to provide high ratings but low reports) in California contributed to understating performance in California relative to the rest of the nation (Weech-Maldonado et al. 2004
3. Are there plan-specific differences in CAHPS scores that indicate that unique plan-level characteristics might be affecting observed California/non-California differences in scores?
CAHPS performance in California was being affected by organizational and operational factors operating among the MA plans that do not exist in the fee-for-service Medicare sector. In particular, a single California health plan substantially altered the average MA CAHPS scores for California. This plan had strong scores for all the CAHPS measures except physician services, which resembled the national MA average. When its data were removed from the CAHPS data for California MA plans, the average scores for the remaining plans dropped substantially, so they were below the national MA average on all but Part D domains. Further, their differences from national MA means for physician service domains were larger than the differences in the fee-for-service sector; this was not the case for the other CAHPS domains. Thus, one cannot simply conclude that all MA plans in California perform poorly in patient experience of care. Rather, it is important to look within the group at individual health plans to better characterize patterns of performance across plans. The existence of a large plan with consistent above-average performance limits the extent to which unmeasured factors specific to California's population are likely to explain observed differences.
Clinical care processes have been found to vary across individual health plans or across types of plans, and it is reasonable to expect that similar variations would be occurring for patient experience of care. For example, differences in inpatient care utilization have been found for beneficiaries with severe chronic diseases who are in fee-for-service Medicare versus health plans (Revere and Sear 2004
), as have variations across Medicaid health plans in pediatric asthma care (Dombkowski et al. 2005
). Health plan ownership has been found to be associated with risk sharing processes, utilization of hospital inpatient care, catastrophic case management, and drug formularies (Ahern and Molinari 2001
). Conversely, a study that examined the effects of health plan delivery system on clinical quality and patient experience of care (using CAHPS) found that the type of delivery system used affected many clinical measures, but not the CAHPS measures (Gillies et al. 2006
Other likely sources of differences are plans' varying approaches to working with their contracted medical practices that, in turn, can affect how patients experience the care they receive from physicians in those practices. For example, studies have found that physicians were more likely to change their clinical practices if they received care management tools from a medical group or group/staff model health plan (Haggstrom and Bindman 2007
), that the structure of a health plan is related to the duration of office visits by elderly patients (Hu and Reuben 2002
), and that a health plan's method of paying physicians can affect patients' experiences of care (Scoggins 2002
The available data did not permit detailed examination of the effects of particular organization-specific factors on MA plan performance on CAHPS measures. Further, our analysis focused on Medicare fee-for-service and MA plans in just one state. Additional investigations of a broader set of health plans, both Medicare and commercial, are needed to identify factors that affect variations in performance across the greater health plan population. Both qualitative and quantitative methods may help unravel the dynamics of service delivery within a number of health plans, drawing upon existing theory and published research in the organizational behavior and health service literature.
Overall, our case study results suggest some areas for improvements. The evidence for low immunization rates in California, in both fee-for-service and MA, compared with the rest of the country suggests the need for quality improvement efforts. In addition, our findings on the role of race/ethnicity case mix adjustment on the average differences between California CAHPS scores and those for the rest of the country suggest that consideration be given to adding these adjustors in some contexts. Finally, any examination of variations in performance across Medicare fee-for-service and MA will need to consider variations across MA plans.