In this study we find that the present-on-admission (POA) indicator has a significant impact on some of the AHRQ Patient Safety Indicators (PSI) rates in patients undergoing CABG surgery. The PSIs are one component of the quality toolbox developed by AHRQ to facilitate quality improvement and provide hospitals with the opportunity to benchmark their performance [29
In practice, quality assurance is usually triggered by case reviews and focuses on the perceived failures of individual physicians and providers. Medical errors are attributed to "aberrant mental processes such as forgetfulness, inattention, poor motivation, carelessness, negligence, and recklessness [7
]." Since individual cases selected for examination are often reviewed in isolation, as opposed to being reviewed as part of a cohort of similar cases, the critical role of health care systems in causing medical errors is frequently ignored. Because PSI rates are, by construction, a measure of global hospital performance, they shift the focus of error analysis from the individual provider to the level of the health care system. For example, a high rate of postoperative sepsis after CABG surgery across cardiac surgeons is more likely to improve with better adherence to patient safety practices, such as the use of maximum sterile barriers during catheter insertion or the use of antibiotic-impregnated catheters [32
], than by the act of "disciplining" a single physician. Thus, PSIs may provide the impetus for a hospital's leadership to examine the "latent conditions" that lead to medical errors – production pressure, inadequate staffing, fatigue – and help set the stage for the adoption of a true "systems approach" to reducing medical error and improving health care quality.
The AHRQ PSI rates have the advantage of being based on administrative data, which are collected by virtually all hospitals in computerized form, and thus is readily available at low cost. Furthermore, the availability of the AHRQ PSI software in the public domain provides all hospitals with the opportunity to benchmark and track their PSI rates. However, the use of administrative data to monitor complications also has important limitations. In particular, the under-reporting of complications using ICD-9-CM codes, in addition to variability in coding practices across institutions, raises questions regarding the validity of using ICD-9-CM codes to report complications [33
] and creates concerns that public reporting of PSI rates may unfairly penalize those hospitals with more accurate reporting practices. The primary limitation of this study is the assumption that the POA indicator accurately distinguishes complications from pre-existing conditions. Parker and colleagues [23
] recently examined the accuracy of administrative data from California, using the POA indicator to exclude complications, with a clinical registry for CABG patients. Using the clinical data as the gold standard, the sensitivity of the risk factors in the administrative data ranged between 22% to 95%, with most above 50%. For most risk factors, specificity exceeded 90%. AHRQ has recently released a report summarizing the evidence supporting the value and validity of the POA indicator [19
However, these well recognized limitations of administrative data for error reporting should not prevent individual hospitals or hospital systems from using non-public reports based on the AHRQ PSI to facilitate quality improvement. Despite the inherent limitations of risk-adjustment for "leveling the playing field" [34
], public and non-public reporting of hospital mortality rates have been associated with significant decreases in mortality for cardiac [36
] and non-cardiac surgery [38
Our findings in this study examine one of the known limitations of administrative data for error reporting, namely, the inability of administrative data to effectively distinguish between pre-existing conditions and complications. Despite the fact that the AHRQ PSI were designed to "emphasize specificity over sensitivity", we found significant numbers of false positives for some of the PSIs. The planned expansion of the use of the POA indicator to all Medicare claims, beginning in 2007, could improve the validity of the AHRQ PSIs if the AHRQ algorithm were revised to include the POA indicator. Recent research in the private sector has led to the development of a system to identify Potentially Preventable Complications (PPC) [21
]. By incorporating the POA indicator into its algorithms for the PPC groups, it was possible to expand the number of diagnoses that could be considered complications without sacrificing specificity. This expansion in scope of error monitoring, predicated on the use of the POA indicator to distinguish complications from pre-existing conditions, may be the "next step" in the evolution of the AHRQ PSI. This "next-generation" complication reporting system may provide greater opportunities for reducing medical errors and improving health care quality. Our study, by showing significant number of false positives using the AHRQ PSIs, further reinforces the need for the widespread adoption of the POA indicator which will make it possible for revised PSI systems, such as the PPC system, to be widely adopted.
Two recent studies have investigated the impact of the POA indicator on patient safety events. The first, by Naessens and colleagues [15
], was based on hospital discharges from the Mayo Clinic Rochester hospitals. This study found that after eliminating secondary diagnoses that were present on admission, the overall rate of patient safety events decreased by nearly 50%. The second, by Houchens and colleagues [16
], used data from California and New York State Inpatient Databases to examine the impact of the POA indicator. This study found that three of the 13 PSIs greatly over-estimated the number of patient safety events when information from the POA indicator was not used to differentiate pre-existing conditions from complications. For these three PSIs, there were significant discrepancies between hospital risk-adjusted PSI rates before and after excluding pre-existing conditions. Our study adds to the existing literature by focusing on a single medical condition, CABG surgery, as opposed to basing the analysis on all inpatient admissions. We believe that PSIs will be useful only insofar as they allow physicians and hospitals to identify problems and, then to focus QI efforts for specific hospital departments, as opposed to solely providing hospitals with a global measure of patient safety events. In this light, studies evaluating the validity of the AHRQ PSIs should assess disease-specific performance, in addition to global performance. In addition, our previous work has shown that the extent to which complications are mis-identified as pre-existing conditions varies substantially across patient populations (e.g. CABG, abdominal aortic aneurysm repairs, stroke patients) [22
]. Thus, it is likely that the accuracy of PSIs would also vary across patient groups. The accuracy of the AHRQ PSI in CABG patients may be of particular interest to hospitals seeking to improve CABG outcomes that only have access to administrative data without the POA indicator.
Increasingly, private payers and Medicare are promoting the use of financial incentives to improve the quality of care through pay for performance initiatives. Nationally, over fifty-percent of Health Maintenance Organizations covering greater than 80% of enrolled patients, have pay-for-performance programs in place [39
]. Under the Deficit Reduction Act of 2005, the reduction in hospital Medicare payments to hospitals not reporting quality data will increase five-fold from 0.4 percent to 2 percent, and infectious complications will no longer entitle hospitals to higher reimbursement rates [40
]. However, the actual impact of pay-for-performance on quality is largely unknown [41
], although recent work suggests that financial incentives has a relatively modest effect on adherence to process measures [42
]. Even if financial incentives were found to significantly improve adherence to process measures, recent work suggests that adherence to "best practices" has only a relatively modest impact on risk-adjusted 30-day mortality rates for patients with acute myocardial infarctions (0.6%), heart failure (0.1%) and pneumonia (0.1%) [43
]. In light of the weak association between many processes of care and outcome, direct outcome measures, such as PSI and related measures of adverse events, may have an important role in future efforts to improve health care quality.