Empirical Evaluation of CAHPS® Reporting Composites
Intercorrelations of Items, Composites, and Ratings. Items in the “getting needed care” composite were modestly associated with the scale score, correcting for item overlap (). The item-to-scale corrected correlations ranged from .27 to .42. The four items, as well as the composite measure, were more strongly associated with global ratings of all health care than with global ratings of the health plan, personal doctors, or specialists.
Correlations of CAHPS® Adult Survey Items with Reporting Composite and Global Ratings
The four items in the communication composite were uniformly associated with the scale score, with r 's ranging from .68 to .75. These items were also more strongly correlated with global ratings of all health care than with the other ratings questions. The composite score was more strongly associated with the global rating of all health care (r=.58) than with the global ratings of personal doctors, specialists, or health plans (r 's=.41, .26, and .38, respectively).
The two items that comprise the composite assessing doctor's office staff were positively associated with one another (r=.61). These two items, as the ones in the previously discussed composites, were moderately associated with the global health care rating and weakly associated with the other three global ratings.
The Getting Needed Care composite had similar associations with the global ratings of health care and health plan. Moreover, items in the composite tended to have similar association with the global ratings of health care and health plans (i.e., r 's ranged from .31 to .44). The items related to getting needed care were all moderately associated with the scale score. Item-scale correlations were highest for the item assessing problems getting care that the patient and doctor believed necessary (r=.48).
The three items that form the health plan customer service measure were uniformly and weakly associated with the scale score. These items showed stronger correlations with the global rating of the health plan than with the global ratings of doctors and all health care. As with all of the five composites, the scale score was more strongly associated with the global rating than any of the items that form the composite (r=.62).
Confirmatory Factor Analysis. presents the factor loadings for five-, three-, and two-factor models. Measures of goodness of fit are shown below the table. In the five-factor model, the four items in the Getting Care Quickly composite loaded modestly on the first factor. The lowest factor loading in this composite was for the item assessing how often the consumer waited more than 15 minutes in the doctor's office to see the person he or she came to see. This item was reverse coded so that a positive loading indicates that the respondent was more likely to say that the wait was less than 15 minutes. The item related to problems getting a personal doctor or nurse contributed less to the Getting Needed Care composite (factor 2) than the other three items; the item assessing problems getting necessary care contributed most to this composite.
Each item in the Doctor's Who Communicate Well composite was strongly associated with the third factor. Similarly, the two items in the Helpful /Courteous Office Staff composite were strongly associated with factor 4. The items in the Plan Customer Service composite (factor 5) had factor loadings that were modest in magnitude (from .48 to .55).
The five-factor model fit the data well and had slightly better fit than both the two- and three-factor models. For example, the goodness-of-fit index adjusted for degrees of freedom (AGFI) for the five-factor model was .99, whereas the AGFI for the three-factor model was .96, and the two-factor model was .95. The three-factor model was significantly different from the five-factor model (χ2 difference=37,935; df=2; p<.001). Similarly, the two-factor model was markedly different than the five-factor model (χ2difference=51,628; df=4; p<.001).
In the three-factor model, shown in , all factor loadings for the access to care factor are between .35 and .62 in magnitudes similar to the two factors in the CAHPS® composites. The pattern of the factor loadings for the second factor appears to maintain two groups that match the original CAHPS® composites. For example, the loadings for the items related to office staff (i.e., courteous/respectful and helpful) were both less than .7, while the loadings for the four items related to doctor's communication were all greater than .7. The third factor in this model was identical to the one presented in the five-factor model.
The factor loadings for the two-factor model (see ) indicate items that form the “doctors who communicate” and “helpful/courteous office staff ” composites contribute more to factor 1 than the four items from the getting care quickly composite. The lowest factor loading in this two-factor model was for the item assessing waits in the doctor office more than 15 minutes. The two items related to problems getting necessary care and delays for plan approval contribute more to the second factor than the other seven items. The lowest two-factor loadings for factor 2 were for the item related to plan paperwork (AC37) and for plan written information (AC33).
shows the estimated interfactor correlations for the five-factor model. The first factor representing getting care quickly was more positively associated with the composites related to getting care, communicating with doctors, and doctor's office staff than with the plan customer service composite. The “getting needed care” and “plan customer services” factors were positively associated (r=.72). The estimated correlation between the health plan and provider factors in the two-factor model was .66.
Estimated Correlations among CAHPS® Dimensions
Reliability and Validity of CAHPS® Composites and Ratings
Reliability of Report Composites. contains measures of plan-level reliability and internal consistency. All of the CAHPS® composites have high plan-level reliability. The number of responses needed to achieve plan-level reliability of greater than .70 was highest for the “doctors who communicate well” and “helpful/courteous office staff composites” (i.e., more than 140 responses). Fewer than 90 responses would be needed for the other three composites to achieve adequate plan-level reliability.
Reliability Estimates for CAHPS® Reporting Composites and Global Rating Items
The two composites showing the highest internal consistency were related to how well doctors communicate and the helpfulness and courtesy of the doctor's office staff (Cronbach's alpha of .86 and .75, respectively). The remaining three composites had internal consistency coefficients less than .70, with the plan customer service composite having the lowest internal consistency.
Reliability of Global Ratings
Plan level reliability was high for the four global ratings (). Global ratings of overall care and health plan were more reliable than the ratings of personal doctors and specialists. The two ratings of doctors require more responses per plan to achieve adequate plan-level reliability than the ratings of all health care.
Association between Composites and Ratings
displays standardized regression coefficients from linear models in which the four global rating items were regressed on the CAHPS® composites. The smallest amount of explained variance in the dependent variables was for the global rating of specialty care (R2=.13) and the largest for the rating of health care overall (R2=.56). The Doctors Who Communicate Well composite was the strongest predictor of ratings of specialists (β=.17), personal doctors (β=.29), and health care (β=.38). The two composites assessing ability to get care and get it quickly were weakly associated with global ratings of personal doctors, adjusting for the other composites.
CAHPS® Global Rating Items Regressed on Reporting Composites
The best predictor of the health plan rating was the plan customer service composite (β=.42). The plan customer service composite was weakly and positively associated with the rating of all health care. Getting care quickly, courteous office staff, and getting needed care, were associated with the global ratings of all health care and the health plan in the same direction and at about the same magnitude. For example, the standardized coefficient for the “getting needed care” composite was .20 in the all health care model and .28 in the health plan model. Hence, these three composites provide information related to all health care as well as the health plan rating.
The CAHPS® composites were created to summarize complex information for users of reports about health plan performance, mostly consumers. They were not designed to be internally consistent scales. Nonetheless, the scales tend to be internally consistent. Two of the reporting composites have very high internal consistency and three were lower. However, all five reporting composites displayed impressive plan-level reliability.
The item that most appears to need refinement is the one asking about wait times at the doctor's office. This item was weakly correlated with the “getting care quickly” scale score and the global rating of care. This may be because it is the only negatively worded CAHPS® question. Deleting this item had almost no impact on the internal consistency of the composite (i.e., a Cronbach's alpha of .58 would be observed in a three-item index). Deleting this item also had no effect on the plan-level reliability.
Each of the five CAHPS® reporting composites was positively associated with global ratings of health care and health plans. Furthermore, the correlations among these five reporting composites were moderate to high. It is likely that consumers use experiences with their health professionals when rating their plans or experiences with plans may influence ratings of professionals. Nevertheless, communication of doctors and health professionals was the strongest correlate of consumer ratings of health care and the performance of health plan customer service was the best correlate of consumer ratings of the health plan.
To reduce the amount of information presented to consumers, CAHPS®
survey sponsors (e.g., health plans, employers, or other purchasers) could create two composite measures. The two- and three-factor models displayed reasonably good fit in the confirmatory factor analysis. However, the five-factor model representing the CAHPS®
reporting composites displayed better fit to the data. For example, factor 1 in the two-factor model had six items that loaded greater than .60 and four items that loaded less than .55. Thus, the first factor in this simplified model appeared to represent two domains. Although users may wish to present the results of two reporting composites (i.e., health plans and health professionals), some information about health professional's performance will be masked to consumers. To provide consumers with information about distinct domains of health plan performance, users of CAHPS®
surveys should consider presenting the five reporting composites we examined among privately insured individuals. Considerations also should be given to the extent to which the different aggregation schemes adequately represent variability across plans in average CAHPS®
scores (Zaslavsky et al. 2000
Other examinations of the CAHPS®
surveys propose presenting fewer reporting composites with other measures created directly from the survey (Bender and Garfinkel 2001
) or with information on clinical quality (Zaslavsky et al. 2002
). In some circumstances, consumers might be overwhelmed with five composite measures from the survey presented with performance measures from expanded CAHPS®
surveys or other data sources (e.g., Medicare prevention ratings or HEDIS measures). Hence, fewer CAHPS®
composite measures of health plan performance to consumers might be a necessity. Developers of consumer reports of health plan performance must balance the desire for comprehensive information about distinct characteristics of health plans reported in narrow categories with the necessity of reducing cognitive burden among readers.