|Home | About | Journals | Submit | Contact Us | Français|
Public reporting of risk adjusted outcomes for percutaneous coronary interventional (PCI) procedures has been mandated in New York State for more than a decade. Over that time there has been a significant decline in the unadjusted mortality following such procedures. Massachusetts joined New York in 2003 as only the second state to require case level reporting of every coronary interventional procedure performed. In this review, we explore the differences in the populations reported by the two states, and consider possible risks of public reporting of clinical outcomes following PCI procedures including the risk of increasing conservatism in the treatment of the sickest patients. We offer a conceptual framework to understand the potential risk-averse behavior of interventional cardiologists subject to public reporting, and offer several proposals to counteract this potential deleterious effect of reporting programs.
With the recent publication of hospital specific mortality outcomes of isolated coronary artery bypass surgery (CABG) and percutaneous coronary intervention (PCI), Massachusetts joined a growing number of states in publicly reporting risk-adjusted cardiac outcomes . While there are a number of reasons to enthusiastically support public reporting there are potential risks that should be considered when evaluating the overall benefits and costs of such programs.
The most compelling justification for the public reporting of clinical outcomes is the public’s right to know about the care that they are likely to receive from hospitals and physicians they utilize. Such transparency of information should allow patients to make better informed decisions about their healthcare choices. Implicit in this reasoning is that the public be provided with accurate and meaningful data and that access to care remains unchanged. As will be described below, these goals may be more elusive to achieve than hoped by the designers of public reporting programs.
Beyond the right to know, important potential additional benefits of public reporting include accelerating the adoption of “best practices” from successful medical centers as well as leveraging the scrutiny of performance which typically increases attention to process and quality improvement (the Hawthorne effect) . As a result of mandated reporting efforts, high quality datasets have been established and used for critical outcomes research [3,4,5]. Importantly, public reporting provides accountability and transparency in regards to the quality assurance, and thereby enhances trust between patients, regulators, payors and providers .
Alongside these clear benefits of public reporting, however, there are a several limitations to current programs and concerns regarding unintended consequences of these efforts. Developing optimal data collection instruments and assuring adequate data quality from participating centers are significant challenges. In addition, it has been difficult to develop risk adjustment methods that adequately account for the severity of illness in extremely sick patients [7,8,9]. While these risk-adjustment models have demonstrated excellent discrimination, calibration and goodness-of-fit in the overall patient populations studied, there are concerns that these models do not adequately address the patients at highest risk . Finally, as demonstrated in recent surveys of interventional cardiologists, performing physicians may not fully accept the accuracy of risk adjustment , leading to avoidance of higher risk patients, and providing perverse incentives to perform procedures in lowest risk patient populations.
The first Massachusetts public report of PCI risk-adjusted mortality was released in 2005, for cases performed during 2003, and demonstrated comparable performance by all Massachusetts hospitals, as measured by the “Standardized Mortality Incidence Rate” (SMIR) . The state-wide unadjusted in-hospital mortality following PCI was 1.71% (n=12,657 total PCI cases for the last nine months of 2003). Results from this report indicated that Brigham and Women’s Hospital (BWH) outcomes were within expectations given our case-mix, though higher risk patients appeared to experience slightly worse outcomes compared with statewide averages than the lower risk population (neither difference was statistically significant). Given the impact of the report on external assessment of the quality of care at our institution, we undertook a detailed exploration of the factors associated with mortality following PCI. Between January2003 and December 2005 there were 85 in-hospital deaths out of 5050 patients receiving PCI performed at BWH, with an unadjusted mortality rate of 1.68%. While only 2.69% of patients had presented in cardiogenic shock, these patients represented more than 54% of the mortality following PCI with such patients experiencing a 68-fold increase in the risk of death as compared with patients not presenting in shock.
As shown in table 1, 44.7% of patients who died prior to hospital discharge had at least one severe acute medical condition present before the index PCI procedure that were not accounted for in the data collection instrument used by the state mandated effort (the ACC-NCDR) . Typical examples of such severe acute co-morbidities included advanced malignancy, active infection, acute stroke, peri-operative myocardial infarction following major non-cardiac surgery, and anoxic brain injury.
We sought to examine the mortalities further through detailed review of the clinical record and angiograms in an effort to classify the deaths into 1 of 3 categories; 1) no complication of the procedure thought to have contributed to death, 2) complication of procedure possibly related to patient’s death, and 3) procedural complication materially contributing to the patient’s death. Charts and films were reviewed independently by 2 board certified interventional cardiologists blinded to both the identity of the patient and the performing interventional cardiologist. Determinations of causality were made based on major neurologic, vascular, or cardiac complications occurring during the procedure or during the hospitalization following the PCI. Examples include intracerebral hemorrhage, major vascular complications requiring surgery or leading to hemodynamic instability, coronary complications including dissection and loss of vessel or acute or subacute stent thrombosis. Of the 85 deaths, 11 (13%) were categorized as being related to a complication of the PCI procedure. A further 7 (8%) were determined to be possibly related to the PCI procedure. The majority of deaths (67/85, 79%), however, had no identifiable complication of the procedure which was plausibly related to the patient’s death.
Based on this analysis, we sought to improve the performance of the standard risk prediction model by adding available pre-procedural data elements including: presentation with neurologic compromise following a presenting cardiac arrest, history of malignancy, in-hospital onset of acute coronary syndrome (for example following non-cardiac surgery), and presentation to hospital with sepsis. The revised model was then tested using a backward selection algorithm on a boot-strap developed multivariate risk model utilizing our single center PCI experience since 2005. This analysis of 4,921 consecutive PCI cases demonstrated that the addition of the four additional covariates modestly improved the discrimination of the model, with an improvement in the area under the ROC curve from 0.919 to 0.937. However this improvement was not statistically significant, with a pair-wise comparison for improvement in model discrimination having a p-value of 0.171, despite adequate power (>80%) to detect a difference between the two models.
While inconclusive, this initial analysis suggests that there may be additional value to expanding the existing risk prediction models to include high risk markers available at case presentation. In support of this hypothesis is the evidence from MA which has recently begun to use a composite additional risk factor (“compassionate use PCI”) in the risk adjustment model for MA PCI outcomes for 2006. The “compassionate use” variable was developed to identify uniquely high risk cases which were taken for PCI when the long term prognosis of the patient was unclear to the operator, but when there was a class I indication for emergent revascularization. These included survivors of cardiac arrest with neurologic impairment in the setting of STEMI, use of percutaneous ventricular support systems to facilitate high risk PCI, and survivors of multiple cardiac arrests en-route to the hospital. Implicit in the rationale for using mortality as an endpoint of quality is that deaths are a reasonable surrogate for the overall quality of the care being provided. These data, however, illustrate less than one quarter of all deaths were possibly related to the PCI procedure itself.
Given the complexity and acuity of the patients treated with PCI, our analysis would suggest that overall mortality for a given provider is greatly influenced by the severity of illness of the patients the operator is willing to take to a procedure. We hypothesize that the impact of severity of illness may be underappreciated by current risk models and could significantly impact the estimation of quality of care by reporting agencies. However, the analysis above is based on a single center’s experience, and may not be representative of centers who do not perform within the State’s mortality predication expectation. Nonetheless, it would seem prudent to consider the addition of adjudicated outcomes of whether a death following PCI was either likely or possibly related to the procedure; as unrelated deaths in cases with clear indications for PCI procedures ought not be counted “against” institutions or operators.
Data from the New York PCI registry, reveal that the mortality following PCI has declined from 0.90% in 1997  to 0.58% in 2003 , a reduction of 36% (p<0.001). While some point to this reduction in mortality as a success, it is important to realize that this improvement in outcomes can be attributed to many potential factors. It is possible that the reduction in mortality is attributable to improvements in the quality of care in the state. However, a simultaneous trend toward avoidance of performance of PCI in higher risk patients may have also contributed to the observed reduction in crude mortality in NY State. Evidence supporting this hypothesis includes an observed simultaneous decline in the proportion of patients presenting in cardiogenic shock treated with PCI in New York from 1.21% in 1997 to 0.85% in 2003—a 30% decrease (P<0.001) [13,14]. During this same time period, there has been increasing recognition of the importance of emergent revascularization for these such patients based on landmark clinical trials and supported by consensus guidelines for the treatment of STEMI complicated by cardiogenic shock [15,16]. In addition, national data (NRMI) suggests increasing incidence of cardiogenic shock. The most provocative data comes from a recently published retrospective study of the SHOCK registry comparing the outcomes of patients from NY and non-NY patients . NY state patients presenting with cardiogenic shock were less likely to receive angiography, PCI, or CABG. While there was no significant difference in mortality for patients undergoing revascularization for shock between NY and non-NY patients, in-hospital mortality for NY state sites was 1.5 fold higher for patients not revascularized (p=.013 for interaction of NY Site and revascularization). Further illustrating the difference in treatment patterns, those patients undergoing CABG in NY state, the mean time to CABG was 10 fold higher in NY patients (101.2 hours vs. 10.3 hrs, p<0.001). The authors conclusion from these data were that public reporting has encouraged a risk-averse climate in NY state that has real public health implications.
Likewise, Massachusetts has released the clinical outcomes report covering 2003–2005 for PCI . As was observed in NY, the proportion of patients treated with cardiogenic shock has declined from 2.28% of all PCI cases in 2003 to 1.29% in 2005; a decline of 43% (see Figure 2). This dramatic decline was coincident with the first three years of public reporting of PCI outcomes in the state. This decline in the proportion of patients treated with cardiogenic shock was associated with an observed decline in the overall crude mortality following PCI in Massachusetts from 1.71% to 1.56%.
This evidence is observational, and one can only infer a causal relationship between the public reporting of outcomes and the decline in treatment of the sickest patients in the state. It is important to note that for these comparisons, we used the broadest possible definition of shock from the New York registry; shock prevalence was taken as the sum of patients presenting as hemodynamically “unstable” or in “shock”. This definition is more inclusive than the ACC-NCDR definition used in MA, and therefore would tend to underestimate the differences between the two States in the analysis that follows. It is also important to acknowledge that one cannot calculate the true rates of PCI treatment for cardiogenic shock, as clinical hospital admission rates for this diagnosis by year are not publicly available at present. Therefore, we have chosen to use a surrogate measure, which is the proportion of patients treated with PCI who presented with cardiogenic shock.
To help understand how public reporting may influence clinicians to avoid the highest risk cases, we propose a framework of relative risks and benefits and overall clinical acuity. In figure 3, the incremental health benefit for a patient undergoing PCI is plotted along the horizontal axis, and the risk of the procedure, here considered to be the likelihood of survival to discharge following procedure, is plotted along the vertical axis. As shown in Figure 3, the framework can be divided into four quadrants based on the procedural risk (likelihood of survival) and benefit (incremental health benefit to the patient). Shown in green in Figure 3, is the low risk, high benefit (upper right) quadrant, in which, even in the face of public reporting should provide little disincentive for physicians to perform the procedure. An example of such a case may be an otherwise healthy patient presenting with a non-ST elevation MI, in which case there is substantial health benefit from PCI; while the risk of the procedure is quite low. Conversely, there are some patients in whom the risk is high and the benefit is low, shown in red as the lower left hand quadrant in Figure 3. Such a case might include a patient presenting with sepsis complicated by a non-ST elevation myocardial infarction. The upper left and lower right hand quadrants have less certain tradeoffs in terms of risk and benefit and are represented in yellow.
Figure 4 illustrates a variety of clinical scenarios with their putative locations plotted within this relative risk-benefit framework. The patient with sepsis and ST wave changes is plotted in the lower left quadrant (in red). A series of patients with increasing severity of acute coronary syndromes is plotted in the mid-portion of the framework in purple, with the increasing risk (decreasing likelihood of survival) of the procedure noted along with the increasing incremental health benefits for the patient. At the extreme, the patients presenting in cardiogenic shock have the highest incremental health benefit gains from PCI, while their acute risk is also the highest, with observed post-procedural mortality rates of 30–50%. These cases are shown in green on the plot. At the other end of the clinical spectrum are patients with minimal or no coronary artery disease, who would be at extremely low risk following PCI, but who have no clear clinical benefit from the procedure, either. Inappropriate use of PCI in these low risk but “no benefit” cases, has been observed such as the series of patients in California who had minimal coronary disease who were nonetheless treated with PCI .
Also shown in Figure 4 is the hypothetical migration of case mix away from sickest patients toward lower risk (and lower benefit) patients in the face of public reporting of risk adjusted outcomes – shown as the dashed red arrow. Public reporting of outcomes can be expected, to encourage physicians to favor procedures in which the patients are at lowest likelihood of death, since it has been observed that even sophisticated risk adjustment does little to reduce the tendency of physicians to treat lower risk patients . Exacerbating the probable migration of cases from high risk to low risk are the financial incentives to physicians and medical centers which have been shown to drive increased performance of procedures when capacity for PCI exists , and would tend to favor performing increasing number of procedures in lower risk cases. The tendency to become more conservative in the face of public reporting of risk adjusted outcomes is termed “risk avoidance creep” and may partially explain the observed reduction observed procedures for highest risk patients, despite probable patient benefit. Risk avoidance creep is driven by the concern that public reporting of risk adjusted mortality rates may result in either reduced patient referrals or ultimately a loss of the ability to provide the services at all.
Given the importance of the public’s right to understand the quality of care available, public reporting of risk adjusted clinical outcomes should continue and expand beyond simply risk adjusted mortality. However, recognizing the limitations of public reporting programs and their potential impact on physician behavior, the following principles are proposed in order to maximize the benefits of these programs while minimizing their potential harm.
Currently, outcomes report cards are focused almost exclusively on risk-adjusted mortality and procedural volumes. If physicians do not completely trust the risk-adjustment methods utilized, there is an incentive to reduce the proportion of the sickest patients treated, so as to have a more favorable report card. It is possible to reduce this incentive by providing additional measures of quality in the outcomes report, which would discourage “risk avoidance creep”. Reporting the center-specific proportion of high-risk cases treated including all patients who presented with an ST segment myocardial infarction and, separately, patients in cardiogenic shock should help balance the incentives. Moreover, additional clinical data elements that are likely to confer extreme risk to patients can be collected and incorporated into the risk adjustment models used. For PCI, these should start with the presence of incessant ventricular arrhythmias and the presence of coma in the setting of an acute coronary syndrome (in whom neurologic prognosis is unclear at the time of urgent PCI). Strict and consistent definitions of all data elements as well as mandatory annual audits of participating centers are essential for such a system to be reliable. Consideration of separating out such cases coded (and rigorously audited) as “compassionate use” would allow physicians to continue to treat the sickest of patients without fear that such high risk cases will adversely impact their report cards due to inadequate risk adjustment.
In addition, emphasis on center-specific public reporting, as opposed to physician-specific reports, could mitigate risk-avoidant behavior by shifting focus from a single person to what should be encouraged to be a team approach. Individual operator outcomes could still be collected and outliers referred for more thorough review by professional or state organizations.
While states involved in public reporting have emphasized in hospital mortality, 30 day or longer mortality may be more appropriate and may avoid the problems with accounting for inter-hospital transfers and discharge to nursing homes or hospice. In addition, there is rationale for non-mortality outcomes including vascular or ischemic outcomes which may be useful as indicators of technical proficiency and may have statistical advantages due to higher frequency. Finally, investigation of process measures may also help to illuminate differences in quality. National benchmarks for door to balloon times when treating acute myocardial infarction is an example of such a measure, but also could include discharge on appropriate medical therapy.
The collection of universal, granular, high quality datasets is costly in terms of personnel and information system resources. It must be recognized that programs that diligently collect, review and analyze detailed internal clinical outcomes data expend significant resources. For example, Brigham and Women’s Hospital employs the equivalent of two full time data coordinators as well as a systems administrator for the reporting of outcomes of approximately 1,800 PCI cases per year. This level of infrastructure is required to assure adequate data quality despite comprehensive point of care clinical data collection systems and integration of the clinical outcomes database systems with the hospital information systems.
National consensus guidelines regarding the domains, scope, methods and implementation of public reporting of clinical outcomes should be developed and implemented. Recent recommendations regarding cardiovascular outcomes assessment have been published , but broad adoption and implementation of such standards is needed. Strategies must be implemented for dataset development and evolution, with the goal of achieving data element definitions that are operationally specific, clinically meaningful and as unambiguous as possible. In addition, standardized recommendations regarding optimal statistical methods for risk adjustment and reporting should be promulgated . In addition, guidelines as to how and when to investigate outlier performances, based on severity and timing of divergence from expectations should be developed. A recent report from the Institute of Medicine recommended creating a new board within the U.S. Department of Health and Human Services to coordinate the development of standardized performance measures . It would be logical for such an organization to oversee standards development for public reporting of risk adjusted clinical outcomes.
In addition to standard reporting of risk-adjusted outcomes and procedure volumes, significant attention should be devoted to studying procedure appropriateness and access to care. Assurance that “risk avoidance creep” does not occur, and that inappropriate low-risk patients are not being treated unnecessarily would likely require surveillance of a sample of representative angiograms from all practitioners. Where concerns are raised, a more thorough examination of practice patterns could be instituted. Equally important in the measures of appropriateness would be institution of measures to ascertain the proportion of high risk patients who are not receiving generally indicated procedures. Specifically, collecting clinical data on all myocardial infarction patients treated at an institution, rather than only those who undergo PCI, could provide insight into whether a center is evolving toward a more conservative approach, that may be at odds with it’s mission for comprehensive care of all patients presenting to that institution.
Our analysis of data from the Massachusetts experience with public reporting raises serious concerns about the unintended impact on clinical care. Clinical outcome report cards serve a valuable public health purpose and should continue to expand as the public demands accountability from the healthcare system. However, this analysis raises concern over whether overall mortality is a reliable guide to quality. As practicing interventional cardiologists, it concerns us that we occasionally consider the impact of potential adverse events on our hospital’s outcomes when evaluating the risks and benefits of a procedure for a particularly ill patient. However well-intentioned the practice of public reporting may be, the consequences appear to have had the opposite effect. With the thoughtful adoption of principles such as those outlined above, the significant benefits of public reporting of risk adjusted outcomes may be achieved while minimizing the potentially deleterious impacts of reducing physician willingness to perform procedures on patients who would significantly benefit from them.
This research was supported, in part, by grant R01-LM08142 from the National Library of Medicine of the National Institutes of Health, United States of America.
Author disclosures: The authors have no financial arrangements that may represent a conflict of interest related to the contents of this article.
F. Resnic: St. Jude Medical, Inc.- Medical Advisory Board and Consultant, Abbot Vascular- Medical Advisory Board
F. Welt: Medtronic Inc.- Consultant, Speaker’s Bureau, St. Jude Inc.- Speaker’s Bureau
Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.