|Home | About | Journals | Submit | Contact Us | Français|
Risk-adjusted analyses are critical in evaluating trauma outcomes. The National Trauma Data Bank (NTDB) is a statistically robust registry that allows such analyses; however, analytical techniques are not yet standardized. In this study, we examine peer-reviewed manuscripts published using NTDB data, with particular attention to characteristics strongly associated with trauma outcomes. Our objective is to determine if there are substantial variations in the methodology and quality of risk-adjusted analyses and thus, whether the development of best practices for risk-adjusted analyses is warranted.
A database of all studies utilizing NTDB data published through December 2010 was created by searching Pubmed and Embase. Studies with multivariate risk-adjusted analyses were examined for their central question, main outcome measures, analytical techniques, the co-variates in adjusted analyses, and handling of missing data.
Of 286 NTDB publications, 122 performed a multivariable adjusted analysis. These studies focused on Clinical Outcomes (51), Public Health Policy or Injury Prevention (30), Quality (16), Disparities (15), Trauma Center Designation (6) or Scoring Systems (4). Mortality was the main outcome in 98 of these studies. There were considerable differences in the co-variates used for case adjustment. The three most frequently controlled for co-variates were age (95%), Injury Severity Score (85%) and gender (78%). Up to 43% of studies did not control for the five basic covariates necessary to conduct a risk-adjusted analysis of trauma mortality. Less than 10% of studies used clustering to adjust for facility differences or imputation to handle missing data.
There is significant variability in how risk-adjusted analyses using data from the NTDB are performed. Best practices are needed to further improve the quality of research from the NTDB.
Trauma is a leading cause of morbidity and mortality.1 More than 2.8 million people are hospitalized with injury each year in the United States,2 costing more than $406 billion annually in medical costs and lost productivity.3 In order to improve trauma outcomes, it is imperative to identify relevant deficiencies and discrepancies at all levels throughout the trauma care system. An important development towards accomplishing this goal has been the inception and organization of the National Trauma Data Bank (NTDB)—a centralized national trauma registry created by the American College of Surgeons (ACS) with the mission to provide the trauma community with accessible and “consistent, quality data.” 4
Over the past several years, NTDB data has been increasingly used in trauma research. It is the largest repository of trauma data reported from trauma centers throughout the United States (U.S.), Puerto Rico and Canada. Since its inception in 1989, the NTDB has continuously evolved to become a powerful, structured tool for the trauma community. The main attraction of this registry is its potential to outperform regional and hospital-based trauma registries by the provision of increased statistical power to researchers through a larger sample size. Clinicians have applied derivations from the NTDB data to improve the standard of care delivered to trauma patients,5 while administrators and policy-makers have used data from it to inform decisions about injury prevention and control, workforce and financial resource allocation.6 The development and maintenance of the NTDB has required considerable resources, and more still are being expended to collect and continuously improve its data before it is made accessible to academic and research institutions. However, there are no benchmarks or practice standards on how data from this large registry should be analyzed or used. Information on how researchers have effectively utilized this data repository and how they have grappled with its shortcomings is important, as it may inform future research.
One of the major attributes of the NTDB is that it allows for risk-adjusted analyses, which are critical in evaluating trauma outcomes. However, standardization of analytical techniques or variables employed to predict outcomes appears to be lacking. Creation of such standards is important to improve the quality of publications derived from NTDB data and further enhance its impact. A synthesis of such information would guide researchers on the current trends and advanced analytical methodology being used in trauma research. Such knowledge may also be helpful in the establishment of similar national registries for other disciplines. This paper examines the research arising from the NTDB, with particular attention to characteristics known to have a strong association with outcomes after trauma. Our objective is to determine if there are substantial variations in the methodology and quality of these risk-adjusted analyses and thus, to establish whether there is a need to develop best practices for risk-adjusted analyses from large trauma registries.
A literature search in Pubmed and Embase was performed for all peer-reviewed publications that have used data from the NTDB from its inception through December 2010. The following search terms were used, either alone or in combination, in these databases to extract articles: ‘NTDB’, ‘national trauma databank’, ‘databank’, ‘database’ and ‘trauma’. This search strategy yielded a total of 286 individual articles which were examined in detail. Studies selected were assessed for their central question, main outcome measures, analytical techniques, type and number of covariates used while performing adjusted analyses, and handling of missing data.
In order to study the impact of the NTDB on trauma outcomes, the distinction was made apriori to focus only on studies that used multivariate adjusted analyses to predict a measureable trauma outcome. This strategy was used because the NTDB is not a population-based database, but rather, the only large database with direct measures of patient injury severity. Therefore, its real power rests not in the provision of epidemiological data, but in its ability to perform adjusted analyses of outcomes.
Criteria for the exclusion of reports included at least one of the following: a) The report did not perform adjusted analyses, b) The report did not analyze any outcomes; this included papers describing or comparing statistical models, such as receiver operating characteristic (ROC) etc., c) A database other than NTDB was used and appeared in the literature search due to overlapping terminology, such as ‘national coma databank’ and ‘trauma’ (studies comparing the NTDB to any other database were, however, included in the final analysis), and d) Methodology or ‘how-to-do’ papers, including first-time descriptions of a novel injury scoring system or statistical method. For manuscripts included in the study, the year and journal the paper was published in, as well as all variables used to predict trauma outcomes, were tabulated in a specially created database. In addition, the use of advanced statistical techniques by researchers, such as propensity scoring or clustering, was noted, as were any contemporary methods of handling missing data such as multiple imputation.
In an effort to better delineate the domains of trauma outcomes being targeted in research by trauma investigators utilizing the NTDB, published reports were also categorized into six domains of trauma outcomes research. A consensus panel of at least four physicians with extensive knowledge of trauma literature categorized the reports. Although more than one theme was undercurrent in several reports, the predominant theme, as judged by the examination of the article generally and its keywords and conclusion specifically, was used to categorize the paper into one of the following:
a) Scoring systems, b) Trauma center designation, c) Disparities, d) Quality of care, e) Clinical outcomes, f) Public health policy or prevention
All co-variates used in adjusted analyses by each paper were also tabulated to better understand what variables researchers were controlling for in their analyses. As researchers may have used different variables to account for similar issues such as anatomical or physiologic injury, co-variates were further grouped into 3 additional categories, namely, “Anatomic severity,” “Physiologic severity,” and “Head injury.” The “Anatomic severity” group consisted of the following covariates: ISS, New Injury Severity Score (NISS), Anatomic Profile (AP), Trauma and Injury Severity Score (TRISS), and the AIS. The “Physiologic severity” group consisted of the following co-variates: systolic blood pressure (SBP), Revised Trauma Score (RTS), respiratory rate (RR), heart rate (HR), TRISS, and base deficit (BD). The “Head injury” group consisted of the following covariates: Head AIS, Glasgow Coma Scale (GCS) scores, GCS-Eye (GCS-E), GCS-Verbal (GCS-V), GCS-Motor (GCS-M), and Head CT status. In a subsequent analysis, if a manuscript suggested that it controlled for any of the co-variates listed in the groupings above, it was given credit for adjusting for that category (e.g. if AP was controlled for, the manuscript was considered to have controlled for anatomic severity). Additionally, the authors established apriori a “bare minimum” set of co-variates which should be required to perform a risk-adjusted analysis for mortality. Each of these co-variates are known to impact survival after trauma and are as follows: 1) Patient Age7, 2) Patient Sex8, 3) Any type of anatomic severity9, 4) Any type of physiologic severity10 and 5) Mechanism or type of injury11. This list was used to determine the number of papers that reported adjusted mortality outcomes, yet did not adjust for these essential minimum co-variates.
One hundred and twenty two papers performed adjusted analyses to determine outcomes after trauma and were included in the final analysis. 164 papers were excluded according to the exclusion criteria listed above. Compared to 2003, when only two publications utilized the NTDB, there was a sharp increase in the number of such published reports in 2009 (n=29) and 2010 (n=33) (Figure 1). As depicted in Figure 2, the highest proportion of studies utilizing the NTDB have been published in The Journal of Trauma (n=53; 43.4%). As expected, the majority of research utilizing the NTDB data is being published in surgical journals. However, fifteen papers (12.3%) have been published in non-surgery journals as well.
As shown in Table 1, each report was categorized into one of the six domains of trauma research: a) Scoring systems, b) Trauma center designation, c) Disparities, d) Quality, e) Clinical outcomes, f) Public health policy or prevention.8–129 The most common type of study noted was the study of a clinical outcome.
Multivariate regression analysis was used by 98 papers (80.3%) to model mortality as an outcome; however, there were considerable differences in the co-variates used for case adjustment between studies. Other outcomes examined included functional status after discharge (commonly measured by Functional Independence Measure scores), hospital and intensive care unit (ICU) length of stay, complications, etc. (Figure 3).
The ten most frequently used co-variates for mortality risk adjustment are shown in Figure 4. Age (93%) and ISS (83%) are the co-variates that have been most frequently used by trauma researchers in risk adjustment for mortality. The co-variates were then grouped into three new categories, namely, “Anatomic severity,” “Physiologic severity,” and “Head injury.” Figure 5 includes these new groups of co-variates and displays the ten most frequently used co-variates. It was observed that most authors were using some measure of anatomic severity (90%), physiologic severity (68%), and head injury (63%). However, up to 43% of papers did not control for at least one of the five co-variates thought to be necessary to risk-adjust for mortality outcomes after trauma. Figure 6 describes the number of papers that adjusted for 1, 2, 3, 4 or all 5 co-variates.
The authors universally acknowledged the limitations posed by missing data points in the NTDB. However, there was considerable variability in how this limitation was handled by investigators. Most reports utilized one of four strategies: exclusion of missing data, subgroup sensitivity analysis, simple or multiple imputation, or the “missing indicator” method. The most common strategy employed by authors (>70%) was the simple exclusion of missing data from the final analysis. For example, in one study of the impact of helmets on injuries among riders of all-terrain vehicles, the authors excluded any cases with missing or unknown protective device use.64 Another way of handling missing data was to use subgroup sensitivity analysis to investigate the qualitative or quantitative effect the missing data. Such analyses are restricted to subsets of the sample population in order to determine whether differences in outcomes are dependent on certain baseline characteristics.130 For example, if there is missing data on co-morbidities, some studies may analyze only the sub-group of patients aged 18-40, who are less likely to have co-morbidities. A third method of handling missing data, especially in situations in which data is known to be missing at random, was the use of simple or multiple imputation, a statistical strategy which uses regression models for each variable with missing data to calculate and estimate missing information.131 Oyetunji et al. have demonstrated that multiple imputation is a feasible strategy for handling missing race and insurance data, using data from the NTDB to show that a complete dataset and multiple imputed dataset were qualitatively similar.132 Finally, the fourth method of handling missing data was the use of the “missing indicator method,” a conditional logistic regression approach, which codes the missing data as “unknown” and includes it in the multivariate analysis.133 This strategy and the imputation techniques were used by less than 5% of authors and only in papers published in the last two to three years. Overall, few studies (<10%) described use of clustering, a statistical technique using generalized linear models (GLM) or the generalized estimating equation (GEE) that can assist in adjusting for the inherent differences between the multiple facilities contributing patients to the sample being studied.
This study demonstrates that researchers are increasingly utilizing the NTDB to study multiple domains of trauma research. NTDB data have been the source of approximately 30 peer-reviewed, risk-adjusted outcomes studies per year for the past few years. Most publications (approximately 80%) have relied on mortality as their main outcome measure; however, there are significant differences in the variables and methodologies used by authors to perform risk-adjusted analyses, even though they share the same outcome measure. In fact, more than 40% of studies reviewed did not control for the 5 basic co-variates regarded as necessary to study risk-adjusted outcomes for mortality after trauma. These disparities in study approaches lead to significant variations in the quality of studies generated from the NTDB and suggest the need to develop best practices or guidelines for large database trauma outcomes studies.
The genesis of the NTDB can be attributed to the concerted efforts of the trauma community over the span of several decades.134-137 The forerunner to the current NTDB was established by the American College of Surgeons – Committee on Trauma (ACSCOT) in 1989 with the vision of bringing improvement in “the care of the injured through systematic efforts in prevention, care and rehabilitation.” The NTDB has grown exponentially from 1 million records in 2004 to more than 4 million records currently.6 In 2010, 682 trauma centers submitted data to this trauma registry, including 210 Level I, 220 Level II and 198 Level III or IV trauma centers.6 In fact, it is now a requirement for ACS verification that all Level I and Level II trauma centers submit data to the NTDB. The original data elements of the NTDB were defined in 1995. More recently, the full adoption of the National Trauma Data Standard (NTDS) in 2007 aims to standardize the inclusion criteria, data coding and collection procedures of all data entered into the NTDB. This initiative facilitates the generation of a more homogeneous and standardized national trauma registry database, and its adoption is expected to further strengthen the NTDB by facilitating comparisons between different trauma systems, improving the reliability of national trauma benchmarks and providing a more robust foundation for trauma outcomes research.4
Currently there is no standardized way for investigators to adjust for mortality risk while doing outcomes analyses. This review of published studies suggests that the absence of such standardization or best practices, in turn, limits the interpretability and external validity of scientific findings derived from the NTDB data. Based on these results, we suggest authors follow the three simple, yet key, considerations portrayed in Table 2 to improve the reliability and generalizability of outcome studies generated from NTDB data.
These basic considerations provide an important starting point for improved standardization of NTDB data analysis. Further quantitative evaluation is needed of co-variates that have been shown to independently influence outcomes and whose incorporation would likely improve the precision and accuracy of prediction models. Thus, the next step to improving NTDB data analysis may be to determine a set of best practices based on consensus of statistical methodology and trauma experts on essential variables and analytical techniques. In time, this could lead to publication of standards that could be used by manuscript reviewers to determine the quality and appropriateness of analyses that utilize NTDB data. Such a level of scrutiny would further help clinicians and policy-makers in determining the level of evidence a paper is presenting and the validity of its findings.
In the present review, it was noted that studies have used NTDB data to investigate a wide variety of topics. Clinical outcomes accounted for more than forty percent of the published studies. With its large sample size, the NTDB allows for the study of common problems such as splenic injury and less common injuries such duodenal trauma. While studying outcomes among patients treated at multiple institutions, it is important to adjust for the effect of inter-hospital outcome variations. One way to do this is the use of clustering by facility during multivariable analyses, which has been shown to impact confidence intervals and therefore, the statistical significance of findings.138 Only a small percentage of the total studies in this review described the use of clustering by facility or similar techniques in their methodology. Quality and public health policy papers were also very common, including studies aimed at quantifying complications. Recent work has demonstrated the importance of selecting the appropriate numerators and denominators when making inferences139 about the prevalence of a specific complication or condition while using NTDB data. Fifteen publications described or reported a race- or health insurance-based disparity in trauma outcomes. The size of the NTDB has been instrumental in describing disparities in trauma outcomes and is another example of its utility.
In this review of studies, mortality was the most widely used outcome measure, which is not surprising, as it is a clearly defined, frequently recorded, reliably documented and readily interpretable measure.41,96 It gives little allowance for error in measurement and can be expected to have minimal bias, despite sampling from geographically diverse trauma centers.96 This review of studies utilizing NTDB data demonstrates that the co-variates used to adjust for patient mix while predicting mortality vary widely and lack standardization. Although certain co-variates such as the ISS (or any measure of anatomical injury severity), age and gender are commonly used, even these measures, which are well-known to impact trauma survival, are not adjusted for by a significant proportion of studies. Similarly, physiologic derangement on ED arrival, which is another well-known predictor of trauma mortality, was only considered by approximately 70% of the studies reviewed. Co-variates such as race, insurance status, mechanism and type of injury, which are also known to impact survival, have been used even less frequently. It is unclear why investigators choose to use only some co-variates in their analysis despite the availability of additional equally important variables in the NTDB. It is important to address this concern in future studies, as inclusion of additional variables known to impact trauma outcomes into statistical models will enhance their ability to deliver accurate results.
It should be acknowledged that researchers must balance the advantages of incorporating additional co-variates into outcome prediction models with the complexities of handling missing data in the NTDB. Any of the statistical techniques commonly used by the authors in this review (ie exclusion of missing data, subgroup sensitivity analyses, etc.) may be appropriate for handling missing data; however, it is critical that researchers appropriately adjust for confounders known to impact mortality. Most investigators exclude patients with missing data from their analysis, leading to considerable loss of sample size.54 This approach can introduce bias and impact the sensitivity and confidence with which differences can truly be ascertained. An analysis by Roudsari et al.138 showed that missing data in the NTDB does not appear to be “completely missing at random,” which raises the possibility of bias with the use of the complete case analysis approach. One method to handle missing information in the NTDB that has recently been successfully utilized in trauma research is the imputation of missing data.36 At least a handful of authors have demonstrated the feasibility of this technique in the NTDB including Oyetunji et al.132 and O'Reilly et al.140 In particular, multiple imputation has been used and validated for missing physiologic data in the NTDB141,142 and holds promise as an excellent technique to decrease the impact of missing data on studies that utilize the NTDB.
Other limitations and challenges with the use of the NTDB have been acknowledged by the American College of Surgeons Committee on Trauma (ACSCOT) and authors in an upfront manner in their reports. The reporting of data to the NTDB is a voluntary undertaking by the trauma centers, making it a convenience sample that is not entirely representative of all trauma centers in the U.S.23,24 This, in turn, creates the potential for selection bias.16 However, the enormous size and cumulative presentation of the sample partially compensates for this limitation. “Unmeasured bias” with the use of NTDB may stem from unknown or residual confounders,8 poor charting, and poor data abstraction.19 The “invisible trauma patient”, who is treated in the emergency room and discharged, is not documented in the NTDB.143 Similarly, patients who died at the scene are not captured by the NTDB.19 Misclassification of variables is possible due to inaccuracy in diagnostic coding of injuries.28 The exact indication for various interventions or diagnostic studies is not mentioned in the NTDB; therefore, it has to be deduced from available information. Furthermore, the potential under-reporting of complications by hospitals submitting data to the NTDB is a significant limitation of the database. Results of studies utilizing the NTDB should be interpreted with the caveat that causal relationships can't be determined with absolute certitude.35 The trends of the associations can be appreciated, but remarks about the precise causes underlying such relationships based on the NTDB data should be made with great care.63
The NTDB represents an exciting chapter in the development of trauma registries. It is a powerful tool that is being used by hundreds of researchers who have published findings in almost every important trauma and surgical journal, as well as non-surgical journals. The aim of this review was to present an organized synthesis of available information about the utilization of the NTDB in the past few years with particular reference to its impact on the study of trauma outcomes. Despite some limitations, the NTDB remains the largest available and most complete national trauma registry61 and has become an invaluable and indispensible data resource for trauma outcomes research. It has not only advanced our understanding of the myriad factors that shape the structure of trauma care in the contemporary medical landscape, but has also provided insight into injury quantification, quality of trauma care, clinical outcomes, burden of injury and disparities prevalent in trauma systems.144 The use of the NTDB for outcomes research is expected to increase in the coming years. Lack of standardization in practices, however, may be a limitation for researchers as they attempt to tap the full potential of the NTDB. Best practices for analyzing data are needed to further improve the quality, reliability and interpretability of research from the NTDB and enhance its impact.
Financial support for this work was provided by: National Institutes of Health/ NIGMS K23GM093112-01; American College of Surgeons C James Carrico Fellowship for the study of Trauma and Critical Care (Dr Haider).
Disclosure Information: Nothing to disclose:
Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.