We conducted this study to evaluate whether outcomes derived from administrative databases accurately represent outcomes obtained from retrospectively reviewing medical charts. Both the administrative database and the chart review identified age, preoperative comorbidities, and emergent surgery as risk factors for postoperative complications following colectomy for UC. However, administrative data overestimated the magnitude of the risk for comorbidities and emergent operations as compared to chart review. Differences in risk estimates were in part explained by misclassification errors associated with the administrative database defining the study population, preoperative risk factors (i.e. comorbidity and emergent colectomy) and postoperative outcome (i.e. complications). The administrative database was more accurate at identifying comorbidities active at admission and the most severe postoperative complications; this selective coding likely biased the risk estimates away from the null hypothesis.
Both clinical [22
] and administrative database studies [11
] have identified advancing age, comorbidities, and emergency operations as risk factors for postoperative complications following colectomy and other abdominal surgeries [25
], though few have evaluated the difference between the two methods[29
]. In our analysis both the administrative and chart review data predicted an approximately two-fold increase in complications in those aged
65 compared to age 18–34
years. Close agreement of OR between administrative and chart data was expected because age is objective and reported with near perfect accuracy in both data sets.
The magnitude of effect for emergent operations was greater with the administrative data as compared to chart review. The administrative database was less specific for identifying emergent colectomy, with a high prevalence of false positives. Sensitivity analysis excluding patients with ‘urgent’ codes demonstrated improved specificity suggesting that the code ‘urgent’ is more aligned with an elective, rather than emergent admission. For example, patients electively admitted for an operation occurring within 24
hours of admission were at times coded as ‘urgent’.
Adaptations of the Charlson and Elixhauser comorbidity indices [20
] have been validated for risk adjustment of postoperative morbidity and mortality [31
]. In our analysis, Charlson comorbidities were significantly associated with worse postoperative outcomes; though, the magnitude of effect was greater in the administrative database than chart data. Preferential recording of comorbidities actively managed in-hospital may explain this difference. The sensitivity for most comorbid illnesses was low, but increased when the analysis was restricted to active comorbidities. Our findings were similar to other validation studies that have found poor sensitivity of comorbidity coding[15
] and underreporting of chronic comorbidities not requiring treatment [34
]. Despite the low sensitivity of administrative data, other studies have found that prediction of in-hospital mortality was identical to indices derived from chart review [31
]. Additionally, among patients with multiple postoperative complications physicians may record more comorbidities in the discharge summary to explain the poor outcomes, while this may not be detailed in patients with an uncomplicated postoperative recovery.
The administrative database was 86% accurate in identifying patients with UC undergoing colectomy. Additionally, a small subset (n
15) of UC patients who underwent colectomy were not recorded in the administrative database. Thirumurthi et al.
found the sensitivity of the diagnostic code 556
× for hospitalization of UC was 84% [36
]. Diagnostic coding for UC may be less accurate than for other conditions; for instance, validations of administrative data in patients presenting with heart failure, acute COPD exacerbations, acute coronary syndromes, and subarachnoid haemorrhage have consistently demonstrated PPV of diagnostic codes exceeding 95% [37
]. In UC, the lower PPV may reflect uncertainties in diagnosis, especially from Crohn’s disease and other causes of colitis. A previous study also reported higher PPV (96.1%) for colectomy codes[41
] compared to our findings, although that validation was performed in a cohort of general surgery patients, with a smaller sample size (n
56), and included procedural codes for rectal resections (484, 485, 486). In our study, follow-up procedures such as second stage ileopouch anal anastomosis were commonly misclassified as colectomies.
Administrative data did not reliably identify UC patients admitted with a flare without colectomy when the first three diagnostic positions were searched. Although nearly 80% of admissions with UC coded in the primary diagnostic position represented an acute flare of disease, UC recorded in the second or third diagnostic positions represented an acute flare in fewer than 10% of cases. This misclassification error is evident in the literature, as one study demonstrated strengthening of risk estimates when a sensitivity analysis was conducted to exclude Crohn’s disease patients admitted to hospital with a secondary diagnosis of Crohn’s disease [42
]. Consequently, prior studies using administrative databases have likely overestimated the true hospitalization rate of UC patients admitted for an acute flare of disease when non-primary diagnostic positions were searched.
The validity of postoperative complications in UC has not been reported. In our study, administrative data was 68% sensitive in identifying patients experiencing at least one complication after colectomy. Previous studies have also shown underreporting of complications in administrative data [43
]. Misclassification of postoperative complications contributed to the discrepancy observed between administrative and chart review data. The accuracy of administrative data in coding postoperative complications was correlated to complication severity: sensitivity increased when less severe complications were excluded from the analysis while the specificity decreased. Administrative data poorly identified minor complications (i.e. Clavien I), but captured the more severe and clinically significant postoperative complications. These findings were similar to our comorbidity validation, supporting the notion that administrative databases miss comorbidities and complications that likely have less clinical impact.
Misclassification of post-operative complications was predominantly due to the challenge in differentiating a postoperative complication from a comorbidity or a preoperative in-hospital complication. For example, UC patients who underwent colectomy and were coded for pulmonary embolism were recorded as a false positive if the pulmonary embolism was diagnosed before the colectomy was performed.
Several limitations of our study should be considered. First, the chart review was retrospective and not all clinical information may have been documented in the charts. As we comprehensively reviewed only the current admission, other comorbidities may have been missed. Second, we only had access to administrative codes for the patient’s hospitalization for colectomy; searching prior admissions may have improved the sensitivity of administrative coding, particularly for comorbidities. This provides an area for future study that may be explored in other datasets. Third, variation between reviewers was unavoidable although we attempted to limit inter-observer variability. Fourth, a small portion (2.6%) of UC patients coded for a flare but not colectomy actually underwent colectomy when the chart was reviewed. Conceivably, UC patients who underwent colectomy may not have been coded for either UC or colectomy, though this misclassification error is likely far less than 2.6%. Fifth, our sample size was sufficient to evaluate the overall validity of administrative data, but uncommon comorbidities and complications could not be validated. Similarly, large administrative database studies have the power to stratify comorbidity as a categorical variable (i.e. 0, 1, 2, or ≥3 comorbidities), but the prevalence of multiple comorbidities in our cohort was too low to accurately perform this subgroup analysis. Finally, the administrative database reflects the quality associated with Calgary’s DAD and thus, may not be generalized to other hospitalization databases. However, Calgary’s DAD is comprehensive, has been widely used and validated for health service research[41
], and has demonstrated generalizability in different settings. For example, a recent study demonstrated that Charlson comorbidities predicted in-hospital mortality similarly in Calgary’s hospital DAD as compared to hospitalization databases in France, New Zealand, Japan, Switzerland, and Australia [48
]. Thus, the data from this study should reflect practices and outcomes of other administrative databases and at minimum should motivate others to test the validity of local administrative databases.