|Home | About | Journals | Submit | Contact Us | Français|
With the proliferation of relatively mature health information technology (IT) systems with large numbers of users, it becomes increasingly important to evaluate the effect of these systems on the quality and safety of healthcare. Previous research on the effectiveness of health IT has had mixed results, which may be in part attributable to the evaluation frameworks used. The authors propose a model for evaluation, the Triangle Model, developed for designing studies of quality and safety outcomes of health IT. This model identifies structure-level predictors, including characteristics of: (1) the technology itself; (2) the provider using the technology; (3) the organizational setting; and (4) the patient population. In addition, the model outlines process predictors, including (1) usage of the technology, (2) organizational support for and customization of the technology, and (3) organizational policies and procedures about quality and safety. The Triangle Model specifies the variables to be measured, but is flexible enough to accommodate both qualitative and quantitative approaches to capturing them. The authors illustrate this model, which integrates perspectives from both health services research and biomedical informatics, with examples from evaluations of electronic prescribing, but it is also applicable to a variety of types of health IT systems.
The potential for health information technology (health IT) to improve the quality and safety of healthcare is the primary impetus behind the federal electronic health record (EHR) incentive program.1 2 However, previous research on the effects of health IT on healthcare delivery has had mixed results, with some studies finding improvements and others showing no effect or adverse effects on quality or safety.3–8
Mixed findings such as these may be in part due to the evaluation frameworks that have been used to assess associations between the quality and safety outcomes and the predictor variable—that is, the health IT itself. For example, several of these studies were before–after studies, which examined the outcomes of interest before and after the introduction of a technology. However, it may not be sufficient simply to categorize a study period by whether or not a specific technology was present. For example, two similar healthcare delivery settings with EHRs or computerized provider order entry (CPOE) systems may be very different from each other, because even the same product will be customized with site-specific configuration of features such as order sets and interfaces with other clinical systems. Training and implementation procedures also differ between institutions and time periods. Furthermore, the technology is not only a predictor variable but also a confounder that can interact with other variables. Technology alters clinical workflow, staffing levels, and user perceptions and attitudes; conversely, organizations can customize technologies to support specific organizational priorities, such as quality measurement or patient safety.
Many of these factors may be potential explanations for the observed differences in quality and safety outcomes for health IT. However, unfortunately, we cannot necessarily be sure of the role of any of these factors unless they are measured reliably and validly. We suggest that research on the impact of health IT on the delivery of healthcare will be stronger if potential predictor variables such as these are captured systematically and prospectively during the evaluation process.
In this paper, we outline the Triangle Evaluation Model, an evaluation model designed to capture the dimensions of assessment necessary to explain the quality and safety effects of health IT, and describe examples of how this model has informed our evaluation work.
The rapid acceleration in use of health IT nationwide, fueled by the federal ‘meaningful use’ policy,1 has resulted in an increased desire to understand how these systems are affecting the quality, safety, and efficiency of healthcare across a variety of healthcare delivery settings. In our view, a joint evaluation approach combining informatics and health services research is the most effective way to answer these research questions.
In developing an evaluation model, we reviewed the literature on published studies evaluating the effects of health IT on quality and safety as well as both evaluation and implementation models specific to health IT. We identified excellent guidance from previous evaluation models and implementation researchers about evaluating a number of aspects of health IT, including technical operations,9 diffusion, adoption, and fit,9–13 cognitive effects,14 15 social, organizational, and workflow impacts,9 16–20 and the general concept of ‘information systems success’.21 In addition, we drew from our own experience conducting quality and safety research in the field of health IT, and conducted iterative discussions within our research team (which contains both health services researchers and informatics researchers) about constructs to be measured and potential operationalization of those measurements in the context of our ongoing and planned research studies.
We accomplished this by mapping elements and processes from health IT models on to the dominant theoretical model in health services research, the Donabedian Model, which emphasizes a systems-level perspective on the determinants of healthcare quality.22 23 According to this model, the quality of a system of healthcare can be defined along three dimensions. ‘Structure’ is the system's material, organizational, and human resources. ‘Processes’ are the activities performed by the system and its people, such as healthcare delivery methods. ‘Outcomes’ are the measurable end results, such as mortality, patient health status, and medical error rates. When applying these concepts to health IT, we were influenced not only by the evaluation literature cited above but also by a second theoretical source, sociotechnical theory, which describes how technology is interconnected with social structure.16 17 20 24 Introducing technology into an organization changes both the organization and the technology; there is ‘a process of mutual transformation; the organization and the technology transform each other’.24
We adapted the Donabedian Model by identifying structure and process factors with the potential to affect quality and safety outcomes of health IT. Four structural variables are depicted in figure 1 and described in additional detail in the next section: (1) the technology; (2) the provider using it; (3) the organizational setting; and (4) the involved patient population. We also identified three categories of processes that connect pairs of structural variables: (1) the use of the technology by the provider; (2) the organizational implementation of the technology; and (3) organizational policies affecting providers (table 1).
In developing the model, we identified elements of healthcare structure and processes that should be assessed concurrently with the outcome variables of quality and safety. In addition, we incorporated the sociotechnical perspective that the organization, technology, and users would influence and change each other, especially through the processes. In this section, we describe the constructs that constitute the model, without specifying how they should be assessed. Assessment methods can be selected according to the resources of the researcher and to the research question at hand.
In the Triangle Model, the relevant elements of structure are: (A) the technology; (B) the healthcare organization; (C) the healthcare provider user; and (D) the patients receiving care. In Figure 1, these elements are represented by the three points of the triangle and the central circle.
In order to assess impact, it is first necessary to inventory the functional capabilities that could affect quality or safety. These would include issues such as the usability of the user interface and the availability of clinical decision support, electronic (e)-prescribing, or interfaces with other systems. Hardware issues and system reliability are also relevant to technology performance.
The healthcare provider who uses the system has attributes that may affect quality and safety outcomes, such as years in practice, training, and attitude toward quality improvement. In addition, some provider attributes such as specialty, typing skills, EHR training and experience, and age may influence how much they use the technology.
Organizational mission, resources, and policies affect quality outcomes directly and also influence how well a technology is used to pursue these outcomes. For example, organizations may or may not create usable EHR configurations in patient examination rooms, devote sufficient resources to EHR training, or make good choices about system configuration. Small medical practices are likely to have different resources and needs for health IT than are large medical centers.
An organization that treats sicker patients will perform poorly on patient outcome measures unless comparisons are adjusted for the population's burden of illness. A variety of methods for risk adjustment, such as case mix adjustment and comorbidity indices, have been developed and validated to more appropriately compare quality outcomes across physicians or healthcare organizations.25–27 Other patient-specific characteristics such as health literacy, patient engagement, and attitudes toward health IT may also be relevant.
In the Triangle Model, processes with the potential to affect quality and safety outcomes of health IT connect the points of the triangle.
Only when a technology is used as intended can relevant quality outcomes be expected. It is thus important to assess the actual usage of the relevant features, which is likely to vary at the level of the individual physician according to usability and perceived usefulness,28–30 integration into clinical workflow and task–technology fit,12 31 and training on the system.
Organizational decisions affect which technologies are implemented, system configuration, implementation procedures, and resources allocated to hardware and technical infrastructure, technical support, and training. As recognized by the DeLone and McLean model of information system ‘success’, these organization-level factors affect the quality of the IT system as implemented in a specific setting, which has a strong impact on use as well as user satisfaction.21
Finally, organizational policies, culture, and workflow all have a direct effect on provider activities and the quality-related outcomes of these activities. For example, an organization may opt into a voluntary quality improvement initiative or pursue a care model transformation such as patient-centered medical home accreditation.32
The Triangle Model specifies the predictor variables that should be captured in order to explain quality or safety outcomes of health IT, but it does not specify how these variables should be measured. In different situations, provider usage of a particular EHR feature might be captured by usage logs, by a researcher making field observations, or by self-reported survey. These approaches each have strengths and weaknesses. In some situations, researchers may prefer intensive qualitative studies to produce a rich and in-depth understanding of a particular situation, whereas in others, researchers may exploit data available from the electronic system itself. The study sample size may limit the number of quantitative predictors that can be included in a regression model, whereas resources may limit the amount of qualitative research that can be performed, creating a need to balance the number of quantitative predictors and qualitative ones included in any particular study.
Two examples presented here illustrate how we have used the Triangle Model to inform our research. Although these both pertain to e-prescribing, the model can be applied to a variety of health IT evaluations.
Our prospective, controlled study of a stand-alone e-prescribing technology was one of the first to demonstrate that e-prescribing was highly effective at reducing prescribing error rates in community-based office practices.33 The primary comparison in the study was users and non-users of this e-prescribing system.
In terms of structural variables from the Triangle Model, we first inventoried the features available in the technology. The inventory suggested that this technology did have the potential to reduce prescribing error rates, as it provided clinical decision support with a wide variety of alert types as well as additional reference resources. At the provider level, we controlled for variables that may have affected prescribing error rates, including years in practice, training, and specialty. Among the patient population, we limited inclusion to adults and collected age, gender, and medications. We studied a single independent practice association (organization), all of whom had access to the same e-prescribing technology and received relatively intensive implementation and technical support (organization–technology processes). For provider–technology processes, we did not quantify usage frequency as a continuous variable because all providers were incentivized to use the system for 100% of prescriptions and thus had very high usage rates. Instead, we minimized variability in our dataset by limiting the study to providers who had used the e-prescribing system to write a minimum of 75 prescriptions.
The outcome variable, prescribing errors, was assessed using a rigorously controlled and previously validated manual review process in which research nurses used a standardized methodology to evaluate paper and electronic prescriptions.
The results of this study were striking. Among providers who adopted e-prescribing, error rates decreased from 42.5 to 6.6 per 100 prescriptions; among non-adopters, error rates remained nearly unchanged (37.3 to 38.4 per 100 prescriptions).33
Capturing the structural elements associated with technology, provider, and patient population allowed us to perform appropriate adjustment in the statistical model, and designing the study to control the variability in the remaining structural and process elements simplified the analyses.
In this pre–post study, the outcome of interest was also prescribing errors, but the primary comparison was between use of two different EHR systems, an in-house system that was replaced at the institutional level by a commercial EHR system (Abramson EL, Patel V, Malhotra S, et al; unpublished data).34
We applied the Triangle Model by inventorying the features available in each technology. The locally developed e-prescribing system provided very little clinical decision support, whereas the commercial system provided a wide variety of clinical decision-support alerts and default dosing with the potential to reduce prescribing error rates. At the provider level, we adjusted for demographics and years in practice, and, among the patients, we restricted eligibility to adults and adjusted for age, sex, and insurance status. This study was conducted in a single organization, where the locally developed system was replaced by the commercial system institution-wide, all physicians underwent the required training, and use of the new system was mandatory (organization–technology and organization–provider processes). The study showed that implementation of the commercial system was associated with a marked fall in the rate of prescribing errors in the short term, with a further decrease at 1 year. However, when inappropriate abbreviations were excluded from the analysis, the rate of errors increased immediately after the transition to the new system, and at 1 year returned to baseline.
Concurrently, we sought additional insight into the provider–technology processes through a survey, semistructured interviews, and field observations; this qualitative data collection was performed concurrently with our quantitative data collection. Among other findings, the results suggested that physicians perceived the locally developed system as faster and easier to use, that the clinical decision-support alerts in the new system led to ‘alert fatigue’ and were often over-ridden, and that few users knew how to use system shortcuts to increase efficiency. These findings provided additional insight into the potential reasons behind the observed spike in certain types of prescribing errors during the transition from one system to the other (Abramson EL, Patel V, Malhotra S, et al; unpublished data). By considering these factors in the design of the research, and conducting qualitative and quantitative evaluation simultaneously, we increased the explanatory power of our study.
Research on the effects of health IT may oversimplify complex issues if health IT is treated as a simple categorical variable (present or absent, or before or after). Capturing more detailed predictor variables about the technology, users, and the surrounding context increases the ability to interpret findings and compare studies, while minimizing the need to cite unmeasured variables as potential explanations of results. In this paper, we have proposed a more comprehensive evaluation model specifically designed for studies of the quality or safety effects of health IT. The Triangle Model specifies that research studies should assess structural elements (the technology, the provider using it, and the organizational setting) and process variables (provider–technology processes such as usage, organization–technology processes such as infrastructure support, and organization–provider processes such as quality improvement initiative), and that evaluations should adjust for characteristics of the patient population.
The Triangle Model carries the implication that unmeasured structure and process variables may account for why field studies of the effects of health IT on quality, safety, and efficiency have had mixed results, with some showing the expected improvements, others failing to find any effect, and others revealing adverse effects.3–7
As an example, we can apply the Triangle Model to understanding two high-profile studies of a commercial CPOE system. Although this system had been shown to reduce prescribing errors and adverse drug events,7 Han et al found that the system was associated with a mortality increase in a pediatric intensive care unit,5 whereas Del Beccaro and colleagues found no such association in a similar setting.8 A critical review of these two papers shows that both sets of researchers came up with plausible potential explanations for their results, but since data were not collected in a systematic fashion as part of the evaluation, no definitive conclusions can be drawn.
Han et al, as well as other commentators,35 attribute their findings to a number of variables they did not measure. They describe some aspects of the system that they felt may have presented usability barriers (interrupting provider–technology processes), as well as problems with the hospital's technical infrastructure such as lack of order sets and an overloaded wireless network (organization–technology processes), and perceived negative effects on workflow and communication (organization factors).5 Similarly, Del Beccaro et al list several unmeasured factors that may have played a role at the level of the technology and the organization, as well as in the interactions between them. These included the organization's construction of order sets and perceived emphasis on encouraging good communication processes among healthcare providers.8 Han et al suggested that the effect of the unmeasured factors was to render the CPOE system slower and less reliable than the old paper-based system and thus less safe in critical care; Del Beccaro et al dispute this interpretation and suggest it was based on an underestimate of the true time needed to place paper orders. However, neither study actually measured the speed of ordering or the other factors cited as potential explanations for the differences between the results. We propose that an evaluation of the system conducted under the Triangle Model might have more systematically captured factors such as these that may have contributed to the healthcare quality outcomes, reducing the need for speculation about the causes of the differences.
In the Triangle Model, we have attempted to summarize and categorize elements from other evaluation models while emphasizing the relationship between technology and healthcare quality and safety outcomes. This creates some similarities to some other evaluation frameworks. Like the SEIPS Model of Carayon et al,19 the Triangle Model adapts the Donabedian Model for use in health IT evaluations. However, SEIPS considers primarily healthcare delivery processes, whereas in the Triangle Model, additional processes of interest include the interactions between the individual user and the technology, and between the organization and the technology. This reflects our emphasis on capturing technology usage patterns as potential predictors of quality outcomes. Our focus on individual usage of technology also distinguishes the Triangle Model from another evaluation model, Westbrook's multi-method evaluation model,20 which encourages the study of organizational-level factors.
The Triangle Model is not intended to be a model of diffusion, adoption, or implementation nor a framework to study outcomes such as successful technology adoption, satisfaction, or workflow. Rather, it is designed to guide evaluations that seek to assess the effect of health IT systems on healthcare delivery, specifically the quality and safety of healthcare. It is thus most appropriate for summative evaluations of relatively mature health IT systems with good adoption rates. As others have noted, summative evaluations are less appropriate for systems in development or in the process of implementation.36 An additional limitation of this model is that we have not specified the measurement instruments or the level of measurement for the various predictor variables we have identified. However, we believe that the resulting flexibility may make this model more generalizable and widely applicable than it otherwise would be. Finally, this model has not been formally validated, and it is possible that additional dimensions could be determined to be useful.
This paper proposes a general model for conducting evaluations of the impact of health IT systems on the outcomes of healthcare quality and safety. This model outlines the domains and constructs that should be assessed, but does not specify whether the methods should be quantitative, qualitative, or hybrid. In our experience, we have found value in applying a variety of different methods, sometimes with the purpose of producing rich qualitative data to explain results, and other times taking advantage of the capabilities of electronic systems to obtain quantitative datasets that allow statistical modeling. We have provided illustrative examples from the domain of medication safety in the ambulatory setting, but the model is broadly applicable to a variety of health IT applications. An evaluation approach that integrates perspectives from health services research and biomedical informatics has the potential to capture the quality and safety effects of the health IT systems that are currently transforming the US healthcare system.
Funding: The investigators are supported by the New York State Department of Health (NYS contract number C023699).
Competing interests: None.
Provenance and peer review: Not commissioned; externally peer reviewed.