|Home | About | Journals | Submit | Contact Us | Français|
Two systems measure surgical site infection rates following colorectal surgeries. Center for Medicare and Medicaid Services pay-for-performance initiatives use National Healthcare Safety Network data for hospital comparisons.
Compare database concordance.
Multi-institution cohort study of system-wide Colorectal Surgery Collaborative. The National Surgical Quality Improvement Program requires rigorous, standardized data capture techniques; National Healthcare Safety Network allows five data capture techniques. Standardized surgical site infection rates were compared between databases. Cohen's Kappa coefficient calculated.
National Healthcare Safety Network or National Surgical Quality Improvement Program patients undergoing colorectal surgery.
Standardized surgical-site infection rates
Thirty-day surgical-site infection rates of 3,547 (National Surgical Quality Improvement Program) vs 5,179 (National Healthcare Safety Network) colorectal procedures (2012-2014). Discrepancies appeared: National Surgical Quality Improvement Program database of Hospital 1 (N=1,480 patients) routinely found surgical-site infection rates around 10%, routinely deemed rate “exemplary” or “as expected” (100%). National Healthcare Safety Network data from the same hospital and time period (N=1,881) revealed similar overall surgical-site infection rate (10%), but standardized rates were deemed “worse than national average” 80% of the time. Overall, hospitals using less rigorous capture methods had improved surgical-site infection rates for National Healthcare Safety Network compared to standardized National Surgical Quality Improvement Program reports. The correlation coefficient between standardized infection rates was 0.03 (p=0.88). During 25 site-time period observations, National Surgical Quality Improvement Program and National Healthcare Safety Network data matched for 52% of observations (13/25). Kappa=0.10 (95% CI: -0.1366-0.3402; p=0.403), indicating poor agreement.
Hospitals located in Northeastern United States only.
Variation in Center for Medicare and Medicaid Services-mandated National Healthcare Safety Network infection surveillance methodology leads to unreliable results, which is apparent when these results are compared to standardized data. High quality data would improve care quality and compare outcomes amongst institutions.
Surgeons share a strong interest in improving the quality of patient care and reducing rates of surgical site infection (SSI). SSIs have previously been found to be associated with increased morbidity, length of hospital stay, and overall cost.1-4 Thus, SSI rate is a common metric used in surgical outcomes research to evaluate surgical quality. However, SSI rate after colon surgery is influenced by several other factors, including patient comorbid diseases, presentation and type of illness, surgical approach, and case complexity.5-8 If SSI rate is to be used accurately to assess the quality of care provided by a surgeon or a hospital for colectomy, detailed data on these potential confounding factors must be accurately collected and the evaluator must take that data into account.
Recently, the Center for Medicare & Medicaid Services (CMS) pay-for-performance initiatives mandated that hospitals enter data into the Center for Disease Control (CDC) National Healthcare Safety Network (NHSN) database.9, 10 The data collected in this database was originally intended for identification of broad epidemiologic trends in SSIs across the country. Because the goal was to only assess large SSI trends.11, 12 the data collection methods were not well standardized. Previous studies have found large discrepancies in the SSI rates reported in NHSN data compared to internal audits,13-15 and their utilization of International Classification of Diseases, Ninth Revision (ICD-9) codes to identify surgical cases for inclusion has been found inaccurate due to non-physician coding.16, 17 Nonetheless, this data is now being used to penalize hospitals: if CMS, reviewing NHSN data, deems infection rates “too high,” these hospitals may lose a percentage of their entire CMS hospital revenue.
In parallel, SSI data is collected at many hospitals through the American College of Surgeons National Surgical Quality Improvement Program (ACS NSQIP). The goal of this surgeon-led program is to track surgical complications, including SSIs, on an apples-to-apples basis, so as to guide quality improvement initiatives at these institutions.18
Given the uncertainty as to whether current NHSN guidelines ensure quality data collection, the aim of this study is to 1) assess the current state of data reporting to NHSN databases across a healthcare network system and 2) determine the concordance of NHSN data with the more rigorous, highly standardized NSQIP database. We hypothesize a low level of concordance of standardized surgical site infection rates between these databases; moreover, we think the NHSN system will reveal lower SSI rates in smaller community hospitals compared to large, academic hospitals due to the lack of case adjustment.
This is a multi-institution cohort study comparing SSI data collected through two national databases used to document and report 30-day postoperative SSI rates following colorectal operations: the ACS NSQIP and CDC NHSN databases. Data following colon surgery performed at five hospitals participating in our system wide Partners Colorectal Collaborative between 2012 and 2014 was gathered. This collaborative includes two high volume academic hospitals and three lower volume community hospitals (performing 50 versus 0-15 major colorectal procedures per month, respectively). The hospitals are labeled Hospital 1-5 based on case volume.
All hospitals attempt to collect 100% of colectomy outcomes data for the NSQIP, and colectomy cases are identified by CPT codes. Certain colectomies are excluded in the 100% capture method: cases performed during nursing reviewer vacations (Hospitals 2-5), colectomies performed by gynecologic providers in the year 2012 (Hospital 1-5), colectomies performed due to trauma, due to a complication from a primary surgery (e.g. colectomy with heart surgery within 30 days), or those performed concurrently with another major case (e.g. hepatectomy with colectomy). Thus, approximately 85% of colonic resections are included in the database.
ASCS NSQIP outcomes data is collected prospectively, through a nationally standardized protocol. Trained surgical clinical nurse reviewers query the medical record and monitor patients for 30-days post-discharge through patient phone calls and direct discussions with visiting nurses, outside emergency rooms and offices of primary care. All readmissions and medical care received post-discharge at any facility is captured to document 30-day complications.
The NSQIP database risk adjustment model includes preoperative, intraoperative, and postoperative variables including CPT codes, information about patient complexity, diagnoses, and comorbidities, which are obtained using standardized and concrete data collection methods. The model is risk-adjusted and case mix-adjusted to account for potential confounders.18, 19 Observed/Expected (O/E) SSI Rate Ratios are calculated using a hierarchical multivariable logistic regression model (Bayesian shrinkage or reliability adjustment).
All hospitals have a separate team under the Infectious Disease Department's supervision to collect 30-day postoperative SSI data for the CDC NHSN database, which uses medical ICD-9 billing codes to identify the cases for its denominator. The aim for this database is to include 100% of all cases involving any form of surgical colonic manipulation (both resection and repair). The data is entered into the NHSN database by a staff member with infection prevention experience, such as an RN or MD. In contrast to NSQIP, NHSN does not exclude pediatric patients, operations for trauma, re-operation to address preexisting colonic infection, performance of concurrent cases and/or multi visceral resections, or surgeries performed to address a complication of another surgical procedure.
Data collection for NHSN SSIs is not standardized from hospital to hospital and can be collected using a number of CDC approved techniques: 1) direct examination of patients' wounds during follow-up visits to either surgery clinics or physicians' offices, 2) review of medical records or surgery clinic patient records, 3) surgeon surveys by mail or telephone, 4) patient surveys by mail or telephone with omissions being marked as no infection, or 5) any combination of the above.20 Table 1 describes the method of infection data collection employed by each hospital in the cohort. Hospital 1 was the only hospital to continue rigorous data capture methods similar to the NSQIP.
The NHSN calculates a standardized infection ratio (SIR) utilizing multivariable logistic regression. Collected infection rates are compared to a previous year's baseline period of data as a benchmark. Expected infection rates are adjusted for age, anesthesia, ASA class, duration of surgery, medical school affiliation, bed size, utilization of laparoscopy and wound classification. Variables such as diabetes mellitus, obesity, immunosuppression, cancer, malnutrition, chemotherapy, or other operations/case-mix are not taken into account in the adjustment.
IRB and Partners Colorectal Collaborative approval was obtained. A step-wise data gathering process was used to compare the two databases at the hospital level. Rates of SSI were calculated as number of infection over number of cases identified as colectomy by the database. NSQIP data was used to describe institutional case mix. Cases were matched between the databases using unique patient identifiers and date of surgery. The NHSN cases not identified in the NSQIP database were matched back to their respective CPT codes and/or patient operative notes to obtain granular detail about these operations.
Data gathered by NSQIP and NHSN databases was compared. Descriptive statistics were reported as percentages for categorical variables. Univariate analysis using General Chi Square test was performed to determine SSI differences and case mix differences between hospitals. Spearman correlation between NSQIP and NHSN SSI rates was assessed by comparing O/E ratios reported by the NSQIP database and SIR reported by the NHSN database. A Cohen's Kappa coefficient was obtained to compare reliability of these two SSI endpoints. Analysis was performed using STATA version 12 (College Station, TX: StataCorp LP).
Hospitals 1-5 reported their 2012-2014 SSI rates following 3,547 colectomies identified by NSQIP and 5,179 colon procedures identified by NHSN, see Figure 1 for cohort description. A careful review of the case mix determined by the CPT codes included in the NSQIP database showed that Hospitals 1-5 performed cases of different complexities, with the overall distribution of right, left, total and other colectomies being significantly different across hospitals, see Table 2. In general, academic hospitals (which were the two highest volume hospitals) performed cases of higher complexity. For example, the higher volume hospitals (Hospital 1 and 2) performed more total colectomies at a rate of 7% compared to 1-2% at smaller volume hospitals (p<0.001). The use of laparoscopy was center dependent ranging from 13% to 61% of cases (p<0.001). In addition, the rate of colostomy/end ileostomy utilization varied by hospital with a range of 12% to 21% of cases (p<0.001). The complexity of cases of each of the participating hospitals is depicted in Figure 2.
More granular data on the types of operations included in the NHSN colonic database and not the NSQIP colectomy database, was obtained. For hospital 1, only 1554 of 1881 cases included in the NHSN denominator (82.6%) were found to be colectomies. 171 colectomy cases that were identified by the NSQIP reviewers were erroneously omitted from the NHSN denominator due to inaccurate ICD-9 coding. NHSN reviewers also incorrectly included 327 cases involving operations miscoded as a colonic case by hospital billers. This included 161 miscellaneous bowel operations (not colectomy or proctectomy by CPT code), 103 proctectomies (which should not have been included due to higher expected complication rates), and 30 procedures involving the small intestine only. Other procedures that were included as colon operations under NHSN at <10 cases over the study period were flap operations, upper gastrointestinal operations, hepatectomy, pancreatectomy, and abdominal explorations. Similar findings were obtained on review of hospital 2; see Table 3 for further details. A few of these additional cases had unusually high SSIs (SSI in 40-100% of cases). 572 NHSN colectomies were not in the NSQIP database and the surgical indication for these procedures was assessed. In Hospital 1, some of these surgical indications included: ischemic bowel following complications caused by major cardiac/vascular surgery, a redo colectomy with end stoma performed 9 days after a patient had a colectomy and an anastomotic leak at an outside hospital, a case using a colonic conduit for an emergent esophageal reconstruction on a patient that ingested lye, various colon repairs following stab wounds or gunshot wounds to the abdomen, and various complex Hartman closure procedures. All of these cases were correctly excluded from the NSQIP database.
The overall number of infections between NSQIP and NHSN was different across all hospitals with infection rates of 12.3% and 8.2%, respectively. Table 4 describes the number of superficial, deep and organ space infections at each hospital as well as the overall number of infections by procedure type. Overall, open colectomy operations had higher infection rates than laparoscopic operations with the highest infection rate occurring in open partial colectomy cases (56.4%). The overall infection rate between NSQIP and NHSN did not match for any of the hospitals with no clear trend in reporting differences. This random variability in infection rate reported was sometimes large (e.g. 3.3% versus 10.7% in Hospital 3). The difference in infection rate between the databases was due to discrepancies in both the denominator (number of colon cases identified) and the numerator (number of infections identified). Table 5 shows that NHSN reports a higher number of colon cases at every hospital with a 40% increase in number of colon cases reported on average. Academic hospitals (1,2) increased the N in their denominator by 27% and 58%, respectively, while community hospitals (3,4,5) increased their N by 52%, 14% and 58%, respectively. Despite higher denominators, the overall number of infections detected by NHSN during the same time interval was lower for every hospital except hospital 1. This resulted in an increase in infection rate in NHSN versus NSQIP for Hospital 1 by 2.3% and a decrease in infection rate for all other hospitals, -9.3%, -6.7%, -7.3%, -1.4%.
Evaluation of the adjusted NHSN and NSQIP reported infection ratings revealed that during the 25 site-time period observations across the collaborative, NSQIP and NHSN ratings matched in only 52% of observations. Twelve of 25 had discordant ratings. For example, Hospital 1 was found to have “exemplary” or “as expected” NSQIP infection ratings (O/E ratios) 100% of the time during each 2012-2014 data collection cycle. During the same time period Hospital 1 was reported to have “Higher” infection ratings 80% of the time from NHSN; see table 6 for further details. There was no correlation between the NSQIP O/E ratio and NHSN SIR with a correlation coefficient of 0.03 (p=0.88), see Figure 3. In addition, there was poor agreement in infection ratings of the two databases across hospitals with a Kappa coefficient of 0.10 (95% CI: -0.14-0.34; p=0.403).
Finally, analysis of the association between SSI rates and case volume for both NHSN and NSQIP found different results. As depicted in Figure 4, the adjusted SSI rate improves in hospitals with higher case volume for the NSQIP database, except for hospital 2. In contrast, the NHSN volume adjusted SIR shown in the same figure reveals a worse infection rate with increasing case volume.
Patients, surgeons and payers will all undoubtedly agree that constant efforts to improve the quality of surgical care should be a priority. Incentivizing surgical quality improvement with public recognition programs and financial rewards may be a reasonable motivator for such change, but these incentives should be based on reliable data. In this study, we compared standardized infection rates reported by NHSN and NSQIP across a healthcare system. Despite the hospitals' adherence to the methodology required by both databases, we found no correlation between adjusted SSI rates from the databases. This means that one of the two databases are potentially providing erroneous data. If NHSN data is unreliable, some hospitals are being incorrectly penalized by CMS for the care they are providing.
The NSQIP data is gathered by surgical teams and is discussed by surgical teams in morbidity and mortality conferences. NSQIP has a track record for obtaining highly reliable data.21 Furthermore, the NSQIP database is well adjusted for patient characteristics as well as surgical complexity18. As such, the NSQIP provides information on surgical outcomes that surgeons trust. NHSN data is gathered by infectious disease nurses without surgical expertise. This database has been consistently criticized for its inaccuracies.13, 14 In this study, we also found that within our Partners Colorectal Collaborative the data gathered for this database can reach diametrically opposed conclusions when compared to data collected to measure surgical quality in the NSQIP database. These differences in reported infection rates and their subsequent conclusions appeared to be non-random. The NHSN database information was different due to a number of factors including: 1) problems with infection identification for the numerator, 2) errors in operation identification for the denominator, 3) inadequate adjustment to standardize infection rates across hospitals with the NHSN adjustment not accounting for case complexity, and 4) inadequate benchmarking for comparisons.
The methods used to identify the number of infections following surgery are inherently different between the two databases. There are five methods of data collection to identify SSIs that the CDC has deemed appropriate for the NHSN database.20 This inconsistency makes the NHSN methods for SSI identification unreliable. This was acknowledged by CDC itself in a recent memorandum.22 Despite this, there have not been any changes to the methods for data collection. Institutions who lose patients to follow-up in the postoperative period or use patient surveys, which are likely to not be completed, to collect SSI data will appear to have low infection rates with NHSN. Given the major differences in the mode of SSI identification between the two databases, it is not surprising that NHSN missed several of the NSQIP identified infections. Hospital one was the only hospital where few infections were missed, and it is the only hospital to use the same rigorous method of data collection for both databases.
Quality data collection also requires reliable identification of colectomy cases. The NHSN database uses ICD-9 codes to identify colectomy cases for inclusion. These codes are assigned by hospital billers with no medical training, and have previously been shown to be inaccurate.16, 17 On our review it resulted in the inclusion of cases like cholecystectomy, which was incorrectly coded as colectomy. In addition, the NHSN database is more inclusive than NSQIP and there was an overall increase in cases by 40% in NHSN. These operations are purposefully excluded from the NSQIP database because they inherently have a higher complication rate. Because these types of operations are likely to be complication outliers, it is not possible to reliably adjust for their characteristics. By including these cases, academic centers are likely to be penalized for caring for high risk patients.
Once a numerator and denominator are determined, assessing the quality of a hospital requires risk adjustment. The NSQIP risk-adjustment model to determine SSI rates takes into account multiple patient and operative variables to ensure that the reflected SSI rate accounts for potential differences when comparing hospital quality. The NHSN model only accounts for age, use of general anesthesia, American Society of Anesthesiologists score>2, duration of surgery, open procedure, lack of medical school affiliation, hospital bed size, and wound classification.23, 24 These variables are unlikely to capture all potential confounders of this outcome. NHSN then compares this SIR with data previously reported to the database. There is no accounting for the urgency of cases including in the benchmark year or changes in case-mix at an institution over time. The outcomes reported by NSQIP and NHSN are so different that hospital 1 was reported to be a high quality performers (with low O/E infection rates) according to NSQIP metrics and a low quality performer with consistently “High” infection rates on NHSN metrics. This is likely due to incomplete adjustment for complex operations, patient factors, and inclusion of emergent cases.
This study should be considered in light of several important strengths and limitations. One strength is that this study includes both academic and non-academic hospitals with varying case volume and complexity. This allowed us to see that the NHSN and NSQIP infection rates did not correlate across different types of hospitals. In addition, because we analyzed data at a single hospital network, we were able to obtain more granular information about the specific operations included erroneously in the NHSN database. Limitations of this study are that it is a relatively small sample of hospitals, and all hospitals are located in a single region. A larger and more diverse sample of hospitals may have allowed us to see other trends that might have been missed.
In conclusion, the variation in NHSN infection surveillance methodology leads to inaccurate results and should be better standardized if they are going to be used for hospitals comparisons and for pay-for-performance initiatives. The inadequacy of these data collection methods are readily apparent when NHSN data are compared to data where these processes are standardized. Alternatively, these databases could be integrated rather than being created in parallel to reduce waste and improve the quality of data collected for the CDC; however, this would require creating a mandate that all US hospitals participate in the ACS NSQIP database or something similar. Regardless of what database we ultimately use for the purpose of hospital level comparisons, surgical health services researchers who understand the need to include operative complexity and the effect of surgical indication when adjusting for infection rates should be involved in improving the methods for measuring surgical quality. High quality, reliable data should be used for performance metrics to inform important policy and financial decisions.
We would like to acknowledge the work done by Alexandra Koffman, Linda Burr, Jill Steinberg, Marybeth McQueen, and Michael Philips to complete this study.
Funding: This work was supported by the National Cancer Institute at the NIH grant R25CA092203 to Dr. Cauley.
This manuscript will be a podium presentation at the American Society of Colon and Rectal Surgeons conference in Los Angeles, CA: April 30th-May 4th, 2016 (podium #S3).
Contributions: All authors mentioned above made substantial contributions to the conception and design of the project, aided in data acquisition, analysis, and interpretation, and provided critical feedback on the content of the manuscript as well as approval of the final version. Authors LB and CEC co-wrote the article in addition to the previously mentioned contributions.