|Home | About | Journals | Submit | Contact Us | Français|
The extent to which patient experiences with hospital care are related to other measures of hospital quality and safety is unknown.
We examined the relationship between Hospital Consumer Assessment of Healthcare Providers and Systems scores and technical measures of quality and safety using service-line specific data in 927 hospitals. We used data from the Hospital Quality Alliance to assess technical performance in medical and surgical processes of care and calculated Patient Safety Indicators to measure medical and surgical complication rates.
The overall rating of the hospital and willingness to recommend the hospital had strong relationships with technical performance in all medical conditions and surgical care (correlation coefficients ranging from 0.15 to 0.63; p<.05 for all). Better patient experiences for each measure domain were associated with lower decubitus ulcer rates (correlations −0.17 to −0.35; p<.05 for all), and for at least some domains with each of the other assessed complications, such as infections due to medical care.
Patient experiences of care were related to measures of technical quality of care, supporting their validity as summary measures of hospital quality. Further study may elucidate implications of these relationships for improving hospital care.
The quality of medical care provided by hospitals is often less than that of optimal (Rosenthal et al. 1998; Jha et al. 2005;). In order to foster improvement, the Hospital Quality Alliance (HQA), a collaboration of leading health care organizations, has identified standard measures of hospital quality that can be used by health care providers to improve quality of care and by consumers to make informed health care choices (Jha et al. 2005; Landon et al. 2006;). To date, the HQA program has focused largely on process measures for three medical conditions and surgery measures pertaining to appropriate prophylaxis for infection and deep venous thrombosis (DVT). Consequently, some have criticized this effort for the relatively narrow spectrum of care encompassed by the measures. For instance, the three medical conditions that HQA focuses on, acute myocardial infarction (AMI), congestive heart failure (CHF), and community-acquired pneumonia, account for just under 10 percent of admissions to U.S. hospitals (Merrill and Elixhauser 2002). Thus, available measures may not adequately reflect overall hospital care quality.
In March 2008, HQA expanded its public reporting efforts to encompass patient experiences of care from the Hospital Consumer Assessment of Healthcare Providers and Systems (HCAHPS) survey (Keeler et al. 1992; Jha et al. 2008;). Because Medicare requires that hospitals collect and report these data in order to receive their full annual payment update, HCAHPS has quickly become the national standard for assessing patient experiences with hospital care. The HCAHPS survey is administered to samples of medical, surgical, and obstetric patients, and it measures nine key aspects of care quality: communication with nurses, communication with doctors, responsiveness of hospital staff, pain management, communication about medicines, discharge information, cleanliness and quietness of hospital, overall rating of the hospital, and patient willingness to recommend the hospital (Kahn et al. 1994; O'Malley et al. 2005;). HCAHPS represents the first national, standardized, publicly reported information that allows consumers to make valid comparisons across hospitals.
Patient-reported measures have several advantages over previously available technical measures. For instance, experiences with care are more easily understood by patients than technical measures. Also, as noted above, available technical process measures have limited scope and coverage, and previous studies indicate that indicators in one domain might not reflect quality of care in other areas (Wilson et al. 2007). Thus, HCAHPS measures, which reflect the experiences of a broader sample of patients, might provide a more representative summary of selected aspects of care quality.
Some, however, may question the importance of patient-reported experiences because they might reflect factors such as a patient's general mood or response tendencies in addition to the actual quality of care. Although case-mix adjustment models have been developed to adjust for such factors, associations between patients' reports of experiences and available measures of hospital quality and safety would add to the credibility of HCAHPS measures. In addition, because many technical measures of quality and safety, such as Agency for Healthcare Research and Quality's (AHRQ's) Patient Safety Indicators (PSIs), have been criticized for their perceived shortcomings (Isaac and Jha 2008), a positive relationship might give the technical measures further credibility as well.
Although previous research has reported on national performance of hospitals with respect to patient experiences, none has separated out performance for the medical and surgical services separately. Because experiences might differ by service, combining reports at the hospital level might obscure or weaken important relationships. We, therefore, examined the associations between HCAHPS scores and HQA processes of care in the medical and surgical service, and selected complications of care as measured by the AHRQ PSIs (Bach et al. 2004). We hypothesized that hospitals where patients reported better experiences of care would have better performance on HQA processes of care and fewer complications as measured by the PSIs.
To examine relationships between HCAHPS and the other hospital quality and safety measures, we compiled data from three data sources. These included 2007 HCAHPS data from over 900 hospitals pertaining to hospital care in 2006 that we obtained from the National CAHPS Benchmarking Database (NCBD), publicly available process of care measures for care delivered in 2006 from the HQA, and PSIs that were calculated using national claims data from the Medicare program in 2005. Our analyses were conducted at the level of the hospital service line (e.g., medicine or surgery), because these are the relevant units for most technical and patient-reported quality measures as well as for patient choice among hospitals (Wilson et al. 2007). The analyses were restricted to hospitals included in the NCBD that also had data available from at least one of the other two data sources.
The NCBD is a voluntary national repository for data from CAHPS surveys. Hospitals that submit HCAHPS data to the NCBD also submit their data to HQA. We used NCBD data rather than publicly available HCAHPS data because the NCBD data are reported separately for each service line (medicine, surgery, and obstetrics) in a hospital rather than pooled together. As patient-assessed hospital quality varies by service line, pooled reporting could obscure significant differences within and across hospitals (O'Malley et al. 2005).
The HCAHPS survey includes 27 questions, 18 of which ask about patient experiences in the areas mentioned above (Kahn et al. 1994). To be eligible for the survey, patients must be at least 18 years old at admission, stay at least one night in the hospital, have a nonpsychiatric principal diagnosis, and be discharged alive. The survey is administered using a standardized protocol 2 days to 6 weeks after discharge by mail, telephone, mail with telephone follow-up, or interactive voice recognition. A random sample of eligible patients is surveyed monthly by the hospital or a licensed vendor and the resulting data are aggregated to produce a rolling 12-month average. Each hospital is directed to obtain 300 completed HCAHPS questionnaires over the year-long reporting period. Small hospitals that are unable to reach the target survey all eligible discharges. The data are adjusted for patient case mix and summarized into individual and composite measures using algorithms developed by the HCAHPS team (O'Malley et al. 2005).
The HQA is a public–private consortium committed to improving hospital care through the collection and public reporting of quality measures. The Medicare Modernization Act of 2003 requires hospitals to submit data on processes of care for all patients with specific medical conditions in order to receive updates on their payment rates from the Centers for Medicare and Medicaid Services (CMS). The HQA specifies specific data collection methods, processes for submitting the data to CMS, and checks for data completeness and accuracy (Medicare.gov—Hospital compare). For medical conditions, we analyzed the 10 core processes of care measures that hospitals are required to submit in order to receive their CMS fee update. For surgery, we examined the measures that are voluntarily reported by hospitals. Scores for each measure reflect the proportion of all patients, including those not covered by Medicare, with one of the conditions who received the specified process of care. We calculated summary performance measures for each medical condition and for surgical care by summing the numerators of related individual measures and then dividing by the sum of the denominators of related measures (Landon et al. 2006). The individual HQA measures used to create the summary performance scores are listed in Table 1.
The PSIs are markers of hospital safety that were developed with the support from AHRQ (Agency for Healthcare Research and Quality 2003). PSI algorithms use administrative data such as ICD-9 codes, demographic characteristics, length of stay, and other data in each discharge to identify preventable complications of inpatient care. The current version includes complication rates that adjust for baseline differences in patient age, gender, modified DRG, and comorbid conditions.
We applied Version 3.1 of the PSI software to the 2005 MedPAR Part A 100 percent file to calculate PSIs related to medical and surgical care. The MedPAR Part A dataset contains discharge data for all hospitalizations of Medicare beneficiaries enrolled in traditional Medicare. We limited our analyses to enrollees of age 65 or older because younger enrollees, who are eligible by reason of disability or end-stage renal disease, may have different conditions and risk factors than elderly enrollees. Because several PSIs have very low incidence rates, we excluded those with no cases in at least half the hospitals, or with a mean rate of less than two cases per 1,000 patients at risk. This limited our analysis to the following medical PSIs: decubitus ulcer, failure to rescue, selected infections due to medical care; and surgical PSIs: postoperative hemorrhage or hematoma, postoperative respiratory failure, postoperative pulmonary embolism (PE) or DVT, and postoperative sepsis. Failure to rescue refers to deaths attributed to specified complications of care during hospitalization such as pneumonia, sepsis, and gastrointestinal bleed. Selected infections due to medical care capture types of vascular access-related infections. Further details about the PSIs, including patient exclusions for each indicator, are available from AHRQ (Patient Safety Indicators Download 2007). We used risk-adjusted complication rates for all analyses.
We excluded hospitals that did not have at least four patients reporting information for particular HCAHPS items or composites within a service (i.e., medical or surgical). We used Fisher's exact test and t-tests to compare the characteristics of study hospitals with all other general medical and surgical hospitals. We did not examine HCAHPS data in the obstetrics service because these are not required by the HQA and few hospitals had sufficient numbers of patients to create stable estimates of technical quality.
We examined the relationship between service-specific HCAHPS scores and the corresponding HQA summary scores. We also examined the relationships of medical PSIs with the HCAHPS composites reported by medical patients, and of the surgical PSIs with HCAHPS composites reported by surgical patients. Bivariate relationships between measures were assessed using hierarchical models that enabled us to estimate correlations among rates net of random sampling error, thus compensating for attenuation of simple correlations due to measurement error. The model we used was of the form θi=(θi1, θi2)′~N(μ,Σ), yij~N(θij, Vij), where θij is the underlying long-term rate or mean for a hospital on a CAHPS measure (j=1) or an HQA or PSI measure (j=2), yij is the corresponding sample estimate with sampling variance Vij, and (μ,Σ) are the mean and covariance matrix of the bivariate θi, respectively. Sampling for yi1 and yi2 was assumed to be independent because the overlap of CAHPS cases with HQA or PSI cases was either nonexistent or minor. We estimated this model for each pair of variables whose correlation was of interest, calculating the estimated correlations as R=σ12/√σ11σ22, where σjk represents an element of Σ. We fitted the hierarchical models using the REML option in SAS PROC MIXED. Besides adjusting for sampling variation, an advantage of this model is that it automatically reduces the influence of the less reliable measures, making it unnecessary to apply ad hoc restrictions to the sample to remove low-reliability hospitals from the estimation process.
Finally, we examined how performance on HCAHPS domains related to performance on HQA and PSI measures. To simplify the presentation, we divided hospitals into quartiles based on performance in each HCAHPS domain and calculated mean performance in each HQA composite measure and each PSI rate.
Of the 927 hospitals in the NCBD hospital dataset, 63 hospitals (6.8 percent) had no information on patient service line. Of the remaining 864 hospitals, 831 had at least one HCAHPS composite score and either medical or surgical HQA data, and 771 had at least one calculable PSI (Table 2). Compared with all general medical and surgical hospitals in the United States, the study hospitals were larger, more frequently for profit, and were more likely to have a cardiac or medical intensive care unit (p<.001). All types of hospitals and regions of the country were represented in the NCBD data.
The means and interquartile ranges for the individual HQA measures in the medical and surgical services are listed in Table 1. Similarly, the means and interquartile ranges for each HCAHPS measure in the medical and surgical service are listed in Table 3. Hospital performance for the AMI and pneumonia composites was strongly associated with patient experiences as measured by HCAHPS (Table 3). For AMI, adjusted correlations were statistically significant for seven of the nine measures. For example, the correlation with the overall rating of the hospital was 0.53 and the correlation with adequate discharge information was 0.43 (both p<.001). In addition, all relationships between pneumonia processes of care and HCAHPS composites were significant, with coefficients ranging from 0.18 to 0.30 (p<.05 for all). Better CHF processes were associated with better overall rating of the hospital and willingness to recommend the hospital (R=0.15, 0.21; p<.05 for both), but not with the other HCAHPS measures. Relationships between surgical processes of care and the HCAHPS measures followed a similar pattern with correlation coefficients ranging from 0.14 for communication with doctors (p=.02) to 0.35 (p<.001) for willingness to recommend the hospital to a friend or family member. Figure 1 illustrates how patients' assessments of willingness to recommend the hospital relate to HQA composite score performance. For example, hospitals in the top quartile of HCAHPS willingness to recommend the hospital to others performed better in AMI processes than hospitals in the worst quartile of HCAHPS rating (89.5 percent versus 84.8 percent).
The means and interquartile ranges for the individual PSIs are listed in Table 1. Adjusted correlations between medical and surgical HCAHPS experiences and relevant PSIs showed a mixed pattern, but significant relationships were generally in the expected direction. In medical patients, decubitus ulcer rates were negatively related with each of the HCAHPS measures with coefficients ranging from R=0.17 for communication with doctors (p=.005) to R=0.35 for responsiveness of the medical staff (p<.001; Table 4). The relationship with infections due to medical care was statistically significant for four HCAHPS measures, including a clean and quiet hospital environment (R=0.37, p<.001), responsiveness of medical staff (R=0.23, p<.001), communication with nurses (R=0.16, p=.01), and communication with doctors (R=0.37, p<.001). There was no relationship between the HCAHPS measures and failure to rescue.
For surgical PSIs, better performance in two PSIs, postoperative respiratory failure and postoperative PE or DVT, were each associated with better performance in five HCAHPS composites (coefficients ranging between −0.33 to −0.44 for the former and −0.15 and −0.21 for the latter; p<.05 for all). The other two PSIs showed few significant relationships with HCAHPS composite performance (Table 5).
When we divided hospitals into quartiles based on performance on each HCAHPS domain, we found relationships with performance on the medical and surgical PSIs consistent with the correlation analyses. For example, hospitals in the top quartile of cleanliness/quietness had fewer selected infections due to medical care than hospitals in the bottom quartile (1.8 events per 1,000 patients at risk versus 2.3 events per 1,000 patients at risk; p<.01 for trend across quartiles). Similarly, hospitals in the top quartile for responsiveness of medical staff had fewer decubitus ulcers than hospitals in the bottom quartile (20.7 events per 1,000 patients at risk versus 25.8 events per 1,000 patients at risk; p<.01 for trend across quartiles).
In this national study of hospitals, examining the relationship between patient experiences and other measures of hospital quality and safety, there were consistent relationships between patient experiences and technical quality as measured by the measures used in the HQA program, and complication rates as measured by the AHRQ PSIs. Two overall measures of hospital performance, the overall rating of the hospital and willingness to recommend the hospital, had strong relationships with better technical performance in processes of care related to pneumonia, CHF, myocardial infarction, and for surgical care. Better patient experiences in all domains were also associated with lower decubitus ulcer rates. Other complications such as infections due to medical care were strongly related to patient experiences in specific domains, such as whether the hospital environment was clean and quiet, and whether the staff was communicative and responsive.
Another study that examined how patients' experiences of care in hospitals related to HQA process measures found similar relationships (Jha et al. 2008). Unlike that study, ours examined patients' perceptions of care stratified by hospital service, which allowed more direct comparison of patient experiences and process-based measures of quality among patients being treated by the same groups of physicians and nurses. Other studies of associations between consumer assessments of care and clinical quality have used data from outpatient care and have yielded mixed results (Schneider et al. 2001; Rao et al. 2006;). For instance, Medicare health plans with better performance on the CAHPS measures had moderately better performance on several Healthcare Effectiveness Data and Information Set measures (Schneider et al. 2001). These relationships may have been weaker than those we found in the hospital setting because there can be significant variation across multiple providers of care within a health plan, and individual health plans might have little direct influence on the quality of care provided by clinicians. Conversely, both survey and technical quality measures for hospitals refer to care provided entirely within a single institution, or in the case of our study, within a service line within an institution.
Moreover, compared with the ambulatory care measures, some of the domains of quality measured by HCAHPS might be more directly related to technical quality of care as measured by either PSIs or HQA. For instance, the cleanliness of the hospital environment was strongly related to prevention of hospital-acquired infections, and responsiveness of medical staff was related to better pneumonia care. Such associations are plausible, though it would be a mistake to infer direct causal links. For example, hospital cleanliness as perceived by patients might not directly reduce infections, but both might reflect a focus on prophylaxis that is manifested in ways that are both visible and invisible to patients. Similarly, patient perceptions of the responsiveness of medical staff might reflect the hospital's safety culture and the adequacy and attentiveness of staff, which might also affect adherence to treatment guidelines and prevention of hospital-acquired infection. The nature of these relationships needs further study to better understand the mechanisms that underlie them.
The HCAHPS measures that have been added to the publicly reported measures of the HQA offer consumers, payers, and policy makers a new perspective on hospital quality. They reflect care provided across virtually all conditions cared for on the medical and surgical services rather than for selected conditions such as AMI or pneumonia, which may be especially salient for patients who are not facing urgent admission. Our finding that patient experiences correlate with technical quality of hospitals enhances the importance of these data, and it suggests that the HCAHPS performance might be a useful overall measure that is broadly reflective of hospital quality.
Our findings also have several implications for quality improvement initiatives within hospitals. First, because performance in processes of medical and surgical care was generally related to multiple HCAHPS domains, efforts to improve processes of care may need to reach several hospital areas and involve both doctors and nurses. Second, because the two general HCAHPS measures, overall hospital rating and willingness to recommend the hospital, had the strongest relationships with processes of care, these measures may be useful adjuncts in assessing the effects of some quality improvement initiatives. Finally, some of these relationships may give quality managers a better understanding of how to reduce certain types of safety complications.
Our study has some potentially important limitations. First, we studied a sample of approximately 800 hospitals from the NCBD rather than all general medical and surgical hospitals that report to the HQA. Although we could have used publicly available data from HQA, the NCBD dataset allowed us to examine the experiences of medical and surgical patients separately, a substantial advantage because our preliminary analyses suggested perceptions of care can vary greatly across service lines within a hospital (O'Malley et al. 2005). Nonetheless, it is possible that the hospitals participating in the NCBD are not representative of hospitals in the United States. Second, each quality measure used has limitations. As mentioned earlier, the HQA measures focus on only a small subset of medical and surgical care. However, the HQA process measures have been widely used as a basis of hospital grading, and CMS and others adopted both of these metrics for payment (Mechanic, Coleman, and Dobson 1998; Ergin et al. 2004;). Third, PSIs rely on accurate and complete coding of complications in billing data, and their validity as safety measures has not been well established (Agency for Healthcare Research and Quality 2003; Zhan and Miller 2003;). Therefore, our finding that better HCAHPS performance is related to lower incidence of some hospital complications should be interpreted with caution, although the correlations in the expected directions are suggestive of a true relationship across the population of hospitals we studied. Fourth, we performed multiple statistical tests examining associations, which create the potential for false-positive results due to chance. However, we found more relationships than would be expected by chance, and many of these relationships were consistent across specific quality and safety measures. Fifth, severity of patients' illnesses in a hospital service may contribute to perceptions of care and outcomes, but both the HCAHPS measures and the PSIs are adjusted for case mix. Finally, the metrics we used for comparison examine different subsets of patients treated in different time periods. Because our analyses were conducted at the hospital level, we think that it is unlikely that there would be significant changes in one or the other measure set over such a short time period. More closely aligned samples might have demonstrated even higher correlations.
In conclusion, the notable associations between measures of patient experiences of care and technical quality and safety in hospitals suggest that HCAHPS measures are important new measures for hospital care quality, even though further study is necessary to elucidate the implications of these relationships for improving hospital care.
Joint Acknowledgment/Disclosure Statement: The authors thank the participants in the NCBD for providing their data, the NCBD staff at Westat for data preparation, and Lin Ding, Ph.D., for statistical programming. This work is supported by grants 2 T32 HP11001-15 from the Health Services Resources Administration and U18 HS13190 from the Agency for Healthcare Research and Quality.
Additional supporting information may be found in the online version of this article:
Appendix SA1: Author Matrix.
Please note: Wiley-Blackwell is not responsible for the content or functionality of any supporting materials supplied by the authors. Any queries (other than missing material) should be directed to the corresponding author for the article.