The practice of medicine is becoming an increasingly data-driven process. Caregivers are required to collect and analyze a large number of variables across many different categories for each patient. Even though methods for studying this data have evolved in recent years, collecting clinical validated data is a cumbersome process. This has resulted in models for acuity adjustment and risk stratification requiring a resource-intensive methodology to create and maintain.
As we study the kinds of data we collect presently, and compare the relative predictive powers of different risk variables in the context of outcomes and risk stratification, opportunities for efficiency and simplification may emerge. In this work we have discussed the issue of how related categories of data only incrementally increase the accuracy of existing risk stratification models. A long-held dictum of the management community has been to seek efficiencies through the 80:20 rule or Pareto principle6
. For example, it is typical that 80% of the commissions at a brokerage are generated by 20% of the brokers. This principle may be true in health care settings as well, where 80% of the risk related to the patient may be modeled using 20% of the data, or maybe 20% of the effort to gather the data. In this sense the incremental benefit of each new variable or new category of variables, each with its own gathering and verification system, can be viewed critically.
In our investigation, we explored questions related to how the accuracy of predictive models is affected by changing the number of variables, the categories of variables, and the times at which these variables were collected. Our results on the NSQIP dataset show that models to predict adverse surgical outcomes can be constructed using fewer variables, with reduced dependence on laboratory results, and potentially using data that is not recorded in the period immediately preceding model training, while still achieving accuracy similar to a more data-intensive approach.
Our findings motivate the creation of acuity models that can be constructed and applied in an affordable and time-efficient manner with low complexity. For example, we note that laboratory results such as albumin levels have been consistently important in the NSQIP dataset while creating models of patient risk. Yet to obtain this one laboratory value at the typical institution would require either a laboratory interface or a separate method for lookup clinically. While albumin has been proven to be valuable in risk stratification, it may be possible to construct predictive models without it that have similar accuracy yet eliminate a level of complexity in the pursuit of quality data that is easier to obtain. In particular, our results on the NSQIP dataset show that patient demographics and clinical characteristics, which can be easily obtained from patient histories and physical exams, contain a wealth of information that can be exploited to reduce dependence on variables that are more invasive and expensive to measure.
Constructing data from easier to collect variables can make the use of acuity models more widespread. Our findings regarding missing data were initially presented as methodologic logistics in an attempt to be fully transparent with data limitations and the characteristics of our study. These findings evolved into further evidence of the difficulties in obtaining reliable data. Even in the NSQIP context with outside audit, dedicated nursing data abstraction and an IRR method, the number of complete datasets is limited. The majority of analytic methods falter in the face of missing data, and methods to extrapolate and recreate these missing values fall short of expectations.
Health care institutions, regulatory and reporting agencies, payers, and even patients increasingly require transparency and reliable outcomes data, preferably with acuity adjustment in a validated way. Health care expenditure is rising rapidly and quality projects struggle to justify expenditure in data gathering and capture. Each variable and additional type or category of variable comes at a greater expense stressing ever-tightening budgets and resources. Methods to improve efficiency without sacrificing accuracy are essential to the continued growth of the quality and outcomes movement in health care.
We conclude by noting that while our study focused exclusively on the NSQIP dataset, the results of our work may extend more broadly to other datasets and clinical disciplines. We also believe that our research on addressing the challenges of collecting data for risk stratification (i.e., time and financial costs, need for invasive tests to measure some parameters) may have further relevance in addressing the burden of cognitive overload. Reducing the number of variables needed to predict patient risk creates the opportunity to identify core data elements that can be compactly presented to caregivers for decision-making. Further research is needed to study both the human factors associated with successfully implementing this approach, and to evaluate the potential impact of such work.