The Triangle Model specifies the predictor variables that should be captured in order to explain quality or safety outcomes of health IT, but it does not specify how these variables should be measured. In different situations, provider usage of a particular EHR feature might be captured by usage logs, by a researcher making field observations, or by self-reported survey. These approaches each have strengths and weaknesses. In some situations, researchers may prefer intensive qualitative studies to produce a rich and in-depth understanding of a particular situation, whereas in others, researchers may exploit data available from the electronic system itself. The study sample size may limit the number of quantitative predictors that can be included in a regression model, whereas resources may limit the amount of qualitative research that can be performed, creating a need to balance the number of quantitative predictors and qualitative ones included in any particular study.
Two examples presented here illustrate how we have used the Triangle Model to inform our research. Although these both pertain to e-prescribing, the model can be applied to a variety of health IT evaluations.
Electronic prescribing improves medication safety in community-based office practices
Our prospective, controlled study of a stand-alone e-prescribing technology was one of the first to demonstrate that e-prescribing was highly effective at reducing prescribing error rates in community-based office practices.33
The primary comparison in the study was users and non-users of this e-prescribing system.
In terms of structural variables from the Triangle Model, we first inventoried the features available in the technology. The inventory suggested that this technology did have the potential to reduce prescribing error rates, as it provided clinical decision support with a wide variety of alert types as well as additional reference resources. At the provider level, we controlled for variables that may have affected prescribing error rates, including years in practice, training, and specialty. Among the patient population, we limited inclusion to adults and collected age, gender, and medications. We studied a single independent practice association (organization), all of whom had access to the same e-prescribing technology and received relatively intensive implementation and technical support (organization–technology processes). For provider–technology processes, we did not quantify usage frequency as a continuous variable because all providers were incentivized to use the system for 100% of prescriptions and thus had very high usage rates. Instead, we minimized variability in our dataset by limiting the study to providers who had used the e-prescribing system to write a minimum of 75 prescriptions.
The outcome variable, prescribing errors, was assessed using a rigorously controlled and previously validated manual review process in which research nurses used a standardized methodology to evaluate paper and electronic prescriptions.
The results of this study were striking. Among providers who adopted e-prescribing, error rates decreased from 42.5 to 6.6 per 100 prescriptions; among non-adopters, error rates remained nearly unchanged (37.3 to 38.4 per 100 prescriptions).33
Capturing the structural elements associated with technology, provider, and patient population allowed us to perform appropriate adjustment in the statistical model, and designing the study to control the variability in the remaining structural and process elements simplified the analyses.
Ambulatory prescribing safety in two EHRs
In this pre–post study, the outcome of interest was also prescribing errors, but the primary comparison was between use of two different EHR systems, an in-house system that was replaced at the institutional level by a commercial EHR system (Abramson EL, Patel V, Malhotra S, et al
; unpublished data).34
We applied the Triangle Model by inventorying the features available in each technology. The locally developed e-prescribing system provided very little clinical decision support, whereas the commercial system provided a wide variety of clinical decision-support alerts and default dosing with the potential to reduce prescribing error rates. At the provider level, we adjusted for demographics and years in practice, and, among the patients, we restricted eligibility to adults and adjusted for age, sex, and insurance status. This study was conducted in a single organization, where the locally developed system was replaced by the commercial system institution-wide, all physicians underwent the required training, and use of the new system was mandatory (organization–technology and organization–provider processes). The study showed that implementation of the commercial system was associated with a marked fall in the rate of prescribing errors in the short term, with a further decrease at 1 year. However, when inappropriate abbreviations were excluded from the analysis, the rate of errors increased immediately after the transition to the new system, and at 1 year returned to baseline.
Concurrently, we sought additional insight into the provider–technology processes through a survey, semistructured interviews, and field observations; this qualitative data collection was performed concurrently with our quantitative data collection. Among other findings, the results suggested that physicians perceived the locally developed system as faster and easier to use, that the clinical decision-support alerts in the new system led to ‘alert fatigue’ and were often over-ridden, and that few users knew how to use system shortcuts to increase efficiency. These findings provided additional insight into the potential reasons behind the observed spike in certain types of prescribing errors during the transition from one system to the other (Abramson EL, Patel V, Malhotra S, et al; unpublished data). By considering these factors in the design of the research, and conducting qualitative and quantitative evaluation simultaneously, we increased the explanatory power of our study.