|Home | About | Journals | Submit | Contact Us | Français|
To describe the developmental process for the CAHPS® Hospital Survey.
A pilot was conducted in three states with 19,720 hospital discharges.
A rigorous, multi-step process was used to develop the CAHPS Hospital Survey. It included a public call for measures, multiple Federal Register notices soliciting public input, a review of the relevant literature, meetings with hospitals, consumers and survey vendors, cognitive interviews with consumer, a large-scale pilot test in three states and consumer testing and numerous small-scale field tests.
The current version of the CAHPS Hospital Survey has survey items in seven domains, two overall ratings of the hospital and five items used for adjusting for the mix of patients across hospitals and for analytical purposes.
The CAHPS Hospital Survey is a core set of questions that can be administered as a stand-alone questionnaire or combined with a broader set of hospital specific items.
The Centers for Medicare and Medicaid Services (CMS) is engaged in a nationwide initiative to make comparative performance information on hospitals available to the public. Such information can help consumers make more informed choices when selecting a hospital and can create incentives for hospitals to improve the quality of care they provide (Hibbard et al. 2005). As part of this effort, CMS has partnered with the Agency for Healthcare Research and Quality (AHRQ) to develop a standardized instrument for measuring patient perspectives on hospital care, known as the Consumer Assessment of Health Providers and Systems (CAHPS®) Hospital Survey.
AHRQ is responsible for developing and maintaining the CAHPS family of instruments for measuring patient perspectives on health care and is a recognized leader in the industry. The CAHPS program develops and supports the use of a comprehensive and evolving family of standardized surveys that ask consumers and patients to report on and evaluate various aspects of their health care (e.g., access to care, interpersonal aspects of care, etc.). CAHPS originally stood for the Consumer Assessment of Health Plans Study, but as the products have evolved beyond health plans, the name has evolved as well to capture the full range of survey products and tools.
Three broad goals have shaped the CAHPS Hospital Survey. First, the survey is designed to produce comparable data on the patient's perspective on care that allow objective and meaningful comparisons among hospitals on domains that are important to consumers. Second, public reporting of the survey results is designed to create incentives for hospitals to improve their quality of care. Third, public reporting will serve to enhance accountability in health care by increasing the transparency of the quality of hospital care provided in return for the investment.
Under the CAHPS I project (1995–2001), AHRQ and its grantees (RAND, Harvard Medical School, and RTI International) developed the original CAHPS instrument for health plans, which is currently used to assess the care provided by health plans covering over 130 million Americans. In 2002 AHRQ funded CAHPS II grantees (RAND, Harvard Medical School, and the American Institutes for Research) to continue and expand the work started in CAHPS I. Westat has served as the coordination and support contractor for both CAHPS I and II.
The CAHPS Hospital Survey initiative focuses on measuring and reporting patients' inpatient experiences with acute care hospitals. Although many hospitals currently use vendor-specific proprietary surveys to collect information on patients' satisfaction with care, there is no standard instrument or data collection methodology that would enable valid comparisons to be made among hospitals that use different survey vendors. To support consumer choice and encourage competition to improve quality of care, it is necessary to make “apples to apples” comparisons based on a standard measurement approach. The CAHPS Hospital Survey is intended to become the standard survey instrument and data collection methodology for measuring patients' perspectives on inpatient hospital care and reporting valid comparisons among hospitals. This methodology and the information it generates will be placed in the public domain for use by hospitals and other interested parties.
Once the survey and implementation strategy are finalized, they will be realized through a voluntary hospital reporting initiative being coordinated by The Hospital Quality Alliance. The alliance includes the major hospital associations, government agencies, consumer groups, measurement and accrediting bodies, and other stakeholders interested in reporting hospital quality. The initial effort invites hospitals to report the results of their performance on 10 clinical quality measures for three medical conditions—acute myocardial infarction, heart failure, and pneumonia. This initial set of clinical measures became available on www.hospitalcompare.hhs.gov in April 2005. Other clinical measures will be added in the future. The CAHPS Hospital Survey scores will be added in 2007.
CAHPS Hospital Survey has been designed to meet CMSs' reporting objectives and the CAHPS design principles. The objectives of the project were to develop (1) the best tool to measure patient perspectives on care for public reporting and (2) a core set of questions to be used by all hospitals, that can be combined with hospital-specific items. As it is designed for public reporting, the CAHPS Hospital Survey is intended to complement, not replace, data currently collected by hospitals for internal quality improvement purposes.
The CAHPS design principles assure that, as a member of the CAHPS family of instruments, the CAHPS Hospital Survey will be reliable, valid, credible, and useful. These principles include:
Because CAHPS instruments, data collection methods, and scoring are comparable among all entities being assessed, the results enable reliable comparisons across sites of care.
When the CAHPS instrument team started to develop the Hospital Survey they wanted to draw on hospital patient surveys currently in use. The development process began in July 2002 when AHRQ published a Federal Register notice soliciting the submission of existing instruments or items measuring patients' perspectives on hospital care, with the understanding that any resulting survey would be placed in the public domain.
Seven submissions were received and carefully reviewed by the CAHPS II grantees to determine if the items captured the patients' perspectives on care in acute care hospitals, demonstrated reliability and validity, and were frequently used. In addition to these criteria, they were judged by the extent to which they reflected the Institute of Medicine's domains of quality of health care, i.e., respect for patients' values; preferences and expressed needs; coordination and integration of care; information, communication and education; physical comfort; emotional support; involvement of family and friends; transition and continuity; and access to care (Institute of Medicine 2001).
In addition to these submitted items, the team reviewed items from the CAHPS adult health plan survey and other related surveys. To further inform its development, the CAHPS team also conducted an extensive review of the literature, focusing on methodological issues such as the extent of standardization of items and methods among hospitals, timing of the survey with respect to discharge date, mode of data collection, response rates, nonresponse adjustment, and case-mix adjustment (Castle et al., 2005).
It was important in the development process to be sure that the type of information which the survey would yield would have value for consumers, who were the primary target for the information. Therefore, focus groups were conducted with consumers for the purpose of identifying which aspects of the hospital experience are most important to them. The results of these focus groups are published elsewhere in this supplement (Sofaer, Crofton, Goldstein, Hoy, and Crabb 2005). The team conducted a total of sixteen focus groups in the fall of 2003 and spring of 2004 in four different cities (Baltimore, Los Angeles, Phoenix, and Orlando). Groups were structured to be homogeneous with respect to type of health insurance coverage (Medicare, non-Medicare) and type of recent hospitalization (urgent admission, elective admission, maternity admission, or no admission). Groups included a mix of men and women of varying race, ethnicity and educational attainment.
To obtain broader input into the development process, the instrument team conducted a public Web chat (October 2002); held meetings for stakeholders (November 2002, November 2003); held a vendor's meeting (November 2002); and established a listserv and e-mail box for continuing public comment. This process produced a draft instrument for cognitive testing.
The draft instrument, originally containing 68 questions, was tested to identify problems with navigation and comprehension in a series of cognitive interviews. Cognitive testing entails administering the survey individually to a sample of persons drawn from the population to be surveyed and asking them questions to determine whether they interpret the items as intended or misunderstand anything about the items. When drafting survey items, researchers often make assumptions about how respondents will interpret the item that prove to be erroneous for persons whose life experience, profession, culture, and literacy differ from their own. Cognitive testing helps to identify these errors so that they may be corrected before the survey is fielded. There were 18 interviews conducted in the first round of cognitive testing, which told us what domains, what terminology, and what response options did not work with the intended audiences. The instrument was revised to address identified problems and tested again with 13 respondents. Of the 31 cognitive interviews, 12 were done in Spanish. Testing was done in Palo Alto, California; Los Angeles, California; Raleigh-Durham, North Carolina; and Boston, Massachusetts (Levine, Fowler, and Brown, 2005). The cognitive testing, combined with field testing, is more likely to yield a survey that is reliable, valid, and the results of which communicate what designers intended.
The final step before pilot testing was to translate the instrument into Spanish. The translation process is described in detail in this supplement in the Hurtado, Angeles, Hays, and Weidmer (2005) article. Briefly, the process involved translation, back translation, and decentering, a procedure in which changes made to improve comprehension in Spanish are used to alter and improve the English version.
This process resulted in a 66-item draft HCAHPS instrument that was submitted to CMS on January 15, 2003. To obtain clearance from the Office of Management and Budget (OMB) for the pilot survey, CMS published a notice in the Federal Register in February 2003 that requested public comments on the draft instrument once more before the pilot survey proceeded.
After obtaining OMB clearance, the 66-item draft was tested as part of the Three-State Hospital Reporting Pilot Study conducted by the Quality Improvement Organizations (QIOs) in Arizona, Maryland, and New York. The three QIOs included Health Services Advisory Group in Arizona, Delmarva Foundation for Medical Care in Maryland, and IPRO in New York. Delmarva coordinated all of the sampling and data collection activities for the pilot. The QIOs acquired the sampling frames from hospitals in their states that either volunteered or were required to participate. All Maryland hospitals were required to participate in the pilot by the Maryland Health Care Commission, while participation was voluntary in the other two states. Delmarva assembled the combined frame and drew the sample. The National Opinion Research Center (NORC) at the University of Chicago collected the data under a subcontract to Delmarva.
The QIO in each state was responsible for recruiting hospitals within its state. A total of 49,812 patients were sampled: 16,431 patients in 45 Maryland hospitals; 11,618 in 26 Arizona hospitals; and 21,763 in 61 New York hospitals.
The sampling frame for the CMS three-state pilot consisted of medical and surgical patients with an overnight stay who were discharged in December 2002 and January 2003 and obstetric patients discharged between November 2002 and January 2003. For 23 hospitals in New York where the sample had to be redrawn, the medical and surgical discharge dates were all in January 2003, and the obstetric discharges were in the December 2002–January 2003 time frame.
Patients excluded from the sampling frame included those under 18 years old at the time of their admission; patients with a psychiatric diagnosis; patients with a discharge status code in categories other than “to home,” patients who died in hospital, obstetric patients whose child died in hospital, and patients with missing data from any fields that were needed for sampling.
We excluded patients discharged to places other than home for two reasons. First, we were concerned that patients discharged to another facility would become confused about which facility to assess or would be influenced in their assessment of the referenced facility by their subsequent experience at a referral hospital, a nursing home, or a post acute facility. We had no prior information about the potential impact of an intervening stay on the scores for the references hospital, so we excluded those patients with the intention of studying this issue in subsequent supplemental field tests that facilities around the country had volunteered to perform.
Second, patients discharged to places other than home are more difficult to locate and more likely to be impaired than patients discharged to home. Persons with more impairment are less likely to respond. Excluding them enabled us to allocate our resources to a larger sample rather than to tracking persons who would be relatively unlikely to respond.
Excluding persons discharged to places other than home may have biased our findings if they have significantly different hospital experiences than other patients and if their responses have different psychometric properties. However, we believe that this potential source of bias is more than off-set by the improved precision produced by larger sample sizes and reduction in potential bias from minimizing the influence of intervening stays.
The sampling objective was to make reliable estimates of CAHPS Hospital Survey scores for each of the medical, surgical, and obstetric service lines within each participating hospital. Our research with stakeholders indicated that hospitals are interested in scores for service lines and we wanted to determine if scores vary systematically by service. Because more hospitals volunteered than initially anticipated and CMS want to allow all interested hospitals to participate, funding was not sufficient to make estimates at the service-line level in each hospital. Thus, CMS assigned all participating hospitals to “core” and “noncore” groups. Core hospitals received a larger sample allocation, to support estimates for the three service lines, and a more intensive nonresponse follow-up effort.
The core hospitals were chosen to ensure that there was a mix of different types of hospitals within each state. Selection of core hospitals was purposive, based on several considerations, including ability to execute the activities necessary to participate in the pilot; number of beds; number of 2001 discharges for medical, surgical, and obstetric patients; average length of stay in 2001 for medical, surgical, and obstetric patients; urbanicity; profit status; and academic medical center status. Steering committees in each state, consisting of representatives from hospital associations and commissions, local hospital administrators, faculty from local medical and nursing schools, state government health care agencies, and consumer advocates, helped select the core hospitals. Pediatric and other specialty hospitals (e.g., cancer facilities, etc.) were excluded from the CAHPS Hospital Survey pilot.
Of the 132 hospitals that participated in the three-state pilot, 24 were designated core hospitals and 108 were designated noncore hospitals. The survey goal for core hospitals was 450 completions, consisting of 150 in each of the three service lines (medical, surgical, and obstetric services). The goal for the noncore hospitals was 150 completions across all three services lines.
To create the sampling frame, each participating hospitals forwarded to their QIO's a data file containing the following items for each eligible discharge: patient name, address, phone number, date-of-birth, and gender; unique patient identifier number; primary discharge diagnosis, disposition upon discharge (e.g., home, skilled nursing facility, etc.); admission and discharge dates; major admission service category (e.g., medical surgical, obstetric); and admission source (e.g., emergency department, skilled nursing facility, etc.). For the core hospitals, a random sample of patients was selected within each of the three service categories (medical, surgical, and obstetrics). A random sample was selected across the three service categories for the noncore hospitals. We assumed a 50 percent response rate; thus, the initial sample for each hospital was twice the target number of completions. The final allocation among service lines in the core hospitals was adjusted to account for the fact that some services did not have the full number of eligible discharges.
Both groups were mailed an advance letter on state-specific letterhead, which referenced the appropriate QIO. The CAHPS Hospital Survey questionnaire, with a cover letter and a return postage-paid envelope, was mailed 1 week later. A reminder postcard was sent to both groups approximately 4 weeks after the first questionnaire was mailed. At this point, the core and noncore methods diverged. Patients in the noncore sample who did not respond to the first mailing were sent a second questionnaire, after which no further follow-up was attempted. Instead of a second mailed questionnaire, nonrespondents from the core hospitals were assigned to telephone follow-up at which computer-assisted telephone interviews were conducted. A maximum of five telephone attempts were made on different days and times of the day to try to maximize response rates.
Data collection began for the core hospitals on June 2, 2003 and ended by August 19, 2003, while for the noncore sample, data collection occurred between June 9, 2003 and October 10, 2003. Survey materials were available in both English and Spanish. The overall response rate was 45 percent for the core hospitals and 35.2 percent for the noncore hospitals. For the core hospitals, 27.4 percent of the responses came from the mailed questionnaire, while the remaining 17.6 percent was from telephone interviews. For the noncore hospitals, 24.4 percent was from the first mailing and 10.8 percent was from the second mailing. Once the data were collected, they were merged with the sampling frame data, cleaned, and edited for errors.
Using hospital administrative data, multiple logistic regression was used to model nonresponse bias. Whites were more likely to respond than other ethnic groups; females were more likely to respond than males; response rates increased with age through 65–74 then declined for those older than 80; having responded to a previous survey was associated with higher response probability; those discharged sick were a little less likely to respond; those who walked out against medical advice were much less likely to respond; those in the hospital two to seven nights were most likely to respond; and response rates varied significantly by DRG (Elliot et al. 2005).
Analysis included evaluation of item–scale correlations (convergence and discrimination), internal consistency reliability for hypothesized multi-item composites, and correlations of items and composites with the global ratings (hospital, doctor, nurses, and whether the patient would recommend the hospital to family and friends (Keller et al. 2005). In addition, case-mix analyses were conducted to identify those variables that are significantly associated with reports and ratings of care (O'Malley, Zaslavsky, Elliott et al. 2005). A variance components analysis was also completed to determine how much of the variation in reports and ratings of care are attributable to regions, hospitals, service category and patients (O'Malley, Zaslavsky, Hays et al. 2005). Other analyses included examination of predictors of unit and item nonresponse as well as characteristics of early versus later respondents (Elliott, Edwards, Angeles, and Hays 2005), and a comparison of the English and Spanish language survey responses (Hurtado et al. 2005).
The analytic sample consisted of 16,045 mail completes and 3,675 phone completes for a total of 19,720 respondents. The sample was approximately evenly divided among core and noncore hospitals (48 and 52 percent, respectively). There was an average of 396 responses per core hospitals and 89 responses per noncore hospital.
We were successful in obtaining a heterogeneous sample of respondents. About 40 percent of respondents indicated that they had been discharged from surgery, 23 percent from obstetrics, and 37 percent from other medical services. Nearly a fourth of the respondents (24 percent) came from Arizona, nearly a third from Maryland (32 percent) and the remainder came from New York. The average length of stay was between three and four nights: nearly 46 percent of respondents had a length of stay of two to three nights; 21 percent stayed one night or less, 24 percent stayed four to seven nights and 9 percent stayed longer than a week. Approximately 36 percent of patients were 18–44 years of age, 27 percent were 45–64 years of age, and 37 percent were 65 years old or older. Because one-third of the sample in core hospitals had to be obstetric discharges, there were twice as many female (66 percent) as male (33 percent) respondents. About 44 percent had a high school diploma/GED or less, and 56 percent had some college or more. Respondents were also racially, ethnically, and culturally diverse: 19 percent described themselves as nonwhite, 10 percent described themselves as Hispanic, and 8 percent spoke a language other than English at home.
Based on the analyses described in this issue, the CAHPS Hospital Survey instrument was reduced to 32 items: 24 items about patients' hospital experiences of care and eight items relating to patients' personal characteristics. The instrument measures seven composites, three global ratings, and one recommendation item. The composites include: nurse communication, nursing services, doctor communication, physical environment, pain control, communication about medicines, and discharge information. The global ratings address nursing care, doctor care and the hospital. The single recommendation item determines if a patient would recommend the hospital to family or friends.
On July 31, 2003, AHRQ and CMS published a Federal Register notice requesting hospitals to volunteer to administer the revised 32-item draft of the CAHPS Hospital Survey, collect the data, and provide that information to AHRQ (Vol. 68, No. 147, pp. 44951–44953). Five hospitals, hospital systems, and hospital reporting cooperatives were selected. The sites selected were the Calgary Health Region, California Institute for Health Systems Performance, California Regions of Kaiser Permanente, Massachusetts General Hospital, and Premier Inc. This additional testing has enabled us to obtain observations from other settings and test design issues that could not be addressed in the three-state pilot, including mode effects and intervening stays. Selected hospitals worked closely with the CAHPS II grantees to design these supplemental studies. They used their own vendors and resources to conduct the survey. AHRQ also conducted additional focus group testing with consumers about the use and value of the CAHPS Hospital Survey results. AHRQ and CMS also made the 32-item instrument available to hospitals, vendors, and other interested parties to voluntarily test the instrument prior to national implementation. A Federal Register notice was published on February 17, 2004 to that effect (vol. 69, no. 31, pp. 7489–7490). This opportunity enabled any vendor, hospital, or other organization to try the CAHPS Hospital Survey instrument, to add the instrument to its own survey, and to evaluate the impact of integrating the CAHPS Hospital Survey into hospitals' current instruments and data collection procedures. We hope that this effort will provide information to further evaluate the methods of data collection prior to national implementation.
A Federal Register notice was published on December 5, 2003 (vol. 68, no. 234, pp. 68087–68088) soliciting input regarding the 32-item version of the survey. As a result of the additional testing conducted and the comments received from this notice, the survey was further refined to 25-items. The 25-item survey was submitted in November 2004 to the National Quality Forum (NQF) consensus development process. NQF is a voluntary consensus standard-setting organization established to standardize health care quality measurement and reporting. Upon the recommendation of the NQF, two items that had been previously deleted from the survey were restored. In May 2005, the 27-item CAHPS Hospital Survey was formally endorsed by the NQF. The NQF-endorsement represents the consensus of many health care providers, consumer groups, purchasers, and research and quality organizations. (See attached appendix for a copy of the 27-item version of the survey.) This version of the survey comprises 22 questions that assess seven key aspects of performance: communication with doctors, communication with nurses, responsiveness of hospital staff, cleanliness and noise level of the physical environment, pain control, communication about medicines, and discharge information. It also includes one overall rating of hospital care and whether the patient would recommend the hospital to friends or family. Five personal characteristics are included to enable adjustment for variation among hospitals in the mix of patients and for subgroup analyses.
Hospitals will voluntarily implement the CAHPS Hospital Survey under the auspices of the Hospital Quality Alliance, a private/public partnership that includes the major hospital associations, government, consumer groups, measurement and accrediting bodies, and other stakeholders who share a common interest in reporting on hospital quality. Once the CAHPS Hospital Survey is full implemented, its results will be reported to the public on the Hospital Compare website, which can be found at www.hospitalcompare.hhs.gov, or through a link on www.medicare.gov. The first full national implementation of the CAHPS Hospital Survey is planned for 2006.
National administration of the survey will be conducted by multiple independent vendors and hospitals using the standard instrument and allowable protocols to ensure that final reported data are sufficiently reliable and valid to permit accurate comparisons among hospitals.
The CAHPS Hospital Survey should be seen as a core set of questions that can be administered as a stand-alone questionnaire or combined with a broader set of hospital-specific items. The CAHPS Hospital Survey is designed to gather only the necessary data that is needed for comparative public reporting and should complement, not replace, data that hospitals are currently collecting to support internal quality improvement activities.
CMS's goal for national implementation is to pursue some level of flexibility in survey administration to allow hospitals and survey vendors to continue with their current survey procedures without sacrificing the comparability of results for public reporting. To that end, there will be a mail version, a telephone version and an active interactive voice response (IVR) version, in which an interviewer initiates the telephone contact. The majority of hospitals and survey vendors use mail or telephone. As CMS is committed to allowing some variation in survey administration protocols, CMS will conduct a large-scale mode experiment in the first year of national implementation to assess the differential impact of allowable administration protocols on the CAHPS Hospital Survey responses and to identify any needed adjustments to assure comparability.
CMS anticipates a decentralized model for the national implementation of the survey with distinct roles for hospitals and survey vendors. Hospitals and vendors will be responsible for all aspects of data collection, including developing a sampling frame of relevant discharges, drawing the sample of discharges to be surveyed, collecting survey data from sampled patients, and submitting CAHPS Hospital Survey data to CMS in a standard format. CMS will be responsible for establishing minimum business requirements for vendors and hospitals choosing to administer the survey themselves, developing a hospital/vendor training and technical assistance program, ensuring the integrity of the data, creating an exceptions process for unique protocols, collecting data from hospitals and vendors, analyzing and weighting the data, conducting a mode experiment to determine an adjustment for differences in data collection modes, and public reporting the data.
The following articles present the results of the analyses of the CAHPS Hospital Survey three-state pilot study data that we have briefly described above. They provide the justification for the 27-item instrument that is presented to the health services research community for scrutiny and comment. As illustrated above, CMS and AHRQ have taken extraordinary steps to make the development process open and public, and the publication of our findings in a highly regarded peer-reviewed scientific journal is the next step in the process. Nevertheless, the process will continue as data for the additional test sites and the national implementation are added to the body of CAHPS Hospital Survey knowledge.
All information that would let someone identify you or your family will be kept private.
You may notice a number on the cover of this survey. This number is ONLY used to let us know if you returned your survey so we don't have to send you reminders.
Please answer the questions in this survey about your stay at the hospital named on the cover. Do not include any other hospital stay in your answers.
Please answer the following questions about your stay at the hospital named on the cover. Do not include any other hospital stays in your answer.
There are only a few remaining items left.
Please return the completed survey in the postage-paid envelope.
The authors would like to acknowledge the support of the Agency for Healthcare Research and Quality and the Centers for Medicare and Medicaid Services for the work on which this paper is based. Work on the CAHPS initiative in general, and the CAHPS Hospital Survey in particular, continuously benefits from the insights and wisdom of all CAHPS teams (American Institutes for Research; Harvard Medical School; RAND Corporation).
Disclaimers: This paper does not reflect any official positions of either the Agency for Healthcare Research and Quality or the Centers for Medicare and Medicaid Services, nor are they responsible for any errors in the paper.