We define public reports as reports that compare measures of provider performance and are accessible to consumers. We include reports that health plans make available to their members, but exclude reports produced by organizations only for internal purposes. We do not include reports that consumers must pay to access. We organize our assessment using three dimensions of public reporting—availability of reports, their credibility and their applicability to their audiences—that have been identified in the literature as related to potential effectiveness. Consumers and providers may view these dimensions differently,12
and the importance of different dimensions may vary with the focus of the report (hospital or physician).
We address availability under the assumption that, if reports are widely available and relatively easy to access, consumers and providers are more likely to be aware of them and the information they contain, a first step to being influenced by the reports. One measure of availability is the number of reports (directed at hospital or physician performance) that exist in any given area. (We do not normalize by population, as we are interested in the availability of information to a given consumer in a study area; use of a report by one person does not preclude use by another.) The larger the number of reports, the more likely that consumers have encountered at least one report and that providers are aware that their performance is being measured and reported. Because most health plan reports are available only to plan members, we also estimate the percent of the population in a given area that has access to plan reports. Finally, we document when reports contain linkages to websites containing similar reports, as this makes additional measures of performance more readily available. In our analysis, we do not assess the efforts of report sponsors to “market” their reports, although this clearly could affect consumer and provider awareness of reports.
We assume that reports with greater credibility are more likely to influence consumer or provider decisions. For consumers, we assume that the primary indicator of credibility is the source of the report, consistent with findings in the literature,11
and that consumers consider health plans to be less trustworthy sources of information, especially when that information is provided online.13
For providers, we assume that credibility is influenced by whether measures in reports are endorsed by respected national bodies or, alternatively, developed locally through a collaborative process involving providers; reports are likely to be viewed as less credible if their measures lack those imprimaturs. Whether reports provide information on patient outcomes, processes of care, or both could influence their credibility with providers, although this could vary with context or provider. For instance, some physicians might regard patient outcome measures for diabetes, such as blood sugar level, as not credible because patient health habits can affect these measures, while others might argue that measures of patient health outcomes increase report credibility. Therefore, in our assessment, we document the presence of each type of measure in public reports.
Reports may be available and credible, but may not contain information that users (providers or consumers) consider applicable to their individual situations. We assume the applicability of reports will affect whether consumers find reports useful in comparing providers. For example, the literature suggests that consumers view patient experience as an important component of quality. 11
Quality of care measures linked to a specific clinical condition are less widely applicable, as they are of value primarily to consumers being treated for that condition. Reports containing multiple measures pertaining to a single condition could be more valuable to consumers with that condition, because they provide more information about different aspects of treatment. However, the literature is not clear on this point, an alternative view being that multiple measures may confuse consumers.11
We document the number of measures in reports that address specific conditions. We also assume that reports will be less applicable if they do not contain information on the type of physician used by consumers.14
Reports that contain only information on primary care physicians, for example, may have more limited applicability than reports combining measures for both primary care physicians and specialists. In addition, the literature suggests that information reported at the physician level is valued more highly by consumers than similar measures constructed at the medical group level;11
we note the number of reports available to consumers that include measures constructed at these different levels.
We assume that the applicability of the contents of public reports to a provider’s specific situation will be important in stimulating provider responses to reports as well. For example, measures of quality that apply only to primary care physicians are not likely to stimulate quality improvement activities by specialists. For hospitals, it is not clear if broadly based measures of hospital quality are more likely to stimulate improvement efforts than measures pertaining to the treatment of specific diagnoses. With regard to the latter, hospital responses could depend on the number of market competitors providing the treatment to which the measure applies. Therefore, we document whether hospital reports contain general versus specific measures.
Our assessment is based on data collected as part of the evaluation of The Robert Wood Johnson Foundation’s Aligning Forces for Quality Initiative (AF4Q).9
Fourteen non-profit organizations (alliances) received initial grants in 2006–2007; three organizations chose to define their entire state as the relevant area for their work, with others focusing on counties or metropolitan areas. Grantees attempt to coordinate and accelerate improvements in quality of care by aligning public reporting, consumer engagement and quality improvement activities. Our analysis utilizes data collected from 7 additional areas (2 states and 5 communities) as well, resulting in 21 study areas in total. The additional areas provide information about the status of public reporting in places without AF4Q grantees and were chosen because of their similarity in location, population size and demographics to one or more AF4Q areas, but are not used for comparison purposes in this analysis.
Tracking Activities We reviewed websites of health plan, hospital and medical associations; quality improvement organizations; state Departments of Health; and AF4Q grantees. In 14 areas, we reviewed websites for the 5 largest health plans, including all national plans. In 7 areas where there were fewer than 5 significant plans, we reviewed the websites for plans whose members together constituted approximately 75 percent or more of the total private sector health plan enrollment. In all 21 areas, we conducted telephone interviews with staff of organizations sponsoring public reports to verify our search findings, gather further details regarding measure sources and their construction, and identify any reports that had not emerged through our search process. We repeated these interviews annually to update our findings, resulting in data that are current through 2009, which we summarize in Tables , , and . (Tables containing detailed information on public reports in specific areas are available online.) Table contains information on report availability across areas, while Table presents indicators of report credibility. The remaining tables address various aspects related to applicability of reports.
Availability of Quality Reports
Credibility of Quality Reports
Applicability of Physician Quality Reports
Applicability of Hospital Quality Reports
Interviews with AF4Q Participants
In the 14 AF4Q areas, we conducted 275 on-site interviews with community stakeholders (employers, providers, health plan representatives, government officials, AF4Q staff) in 2006–2007.15
Subsequently, at 6-month intervals from 2007–2009, we conducted follow-up interviews with AF4Q project directors. Each interview lasted approximately 1 h, was guided by a structured interview protocol and was recorded, transcribed, coded and entered into a database, resulting in approximately 5,500 pages searchable using ATLAS.ti software. We used these interview data primarily in our assessment of the future prospects for public reporting.