PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of nihpaAbout Author manuscriptsSubmit a manuscriptHHS Public Access; Author Manuscript; Accepted for publication in peer reviewed journal;
 
Int J Med Inform. Author manuscript; available in PMC 2011 January 1.
Published in final edited form as:
PMCID: PMC2818499
NIHMSID: NIHMS147317

Social, Organizational, and Contextual Characteristics of Clinical Decision Support Systems for Intensive Insulin Therapy: A Literature Review and Case Study

Abstract

Introduction:

Evaluations of computerized clinical decision support systems (CDSS) typically focus on clinical performance changes and do not include social, organizational, and contextual characteristics explaining use and effectiveness. Studies of CDSS for intensive insulin therapy (IIT) are no exception, and the literature lacks an understanding of effective computer-based IIT implementation and operation.

Results:

This paper presents (1) a literature review of computer-based IIT evaluations through the lens of institutional theory, a discipline from sociology and organization studies, to demonstrate the inconsistent reporting of workflow and care process execution and (2) a single-site case study to illustrate how computer-based IIT requires substantial organizational change and creates additional complexity with unintended consequences including error.

Discussion:

Computer-based IIT requires organizational commitment and attention to site-specific technology, workflow, and care processes to achieve intensive insulin therapy goals. The complex interaction between clinicians, blood glucose testing devices, and CDSS may contribute to workflow inefficiency and error. Evaluations rarely focus on the perspective of nurses, the primary users of computer-based IIT whose knowledge can potentially lead to process and care improvements.

Conclusion:

This paper addresses a gap in the literature concerning the social, organizational, and contextual characteristics of CDSS in general and for intensive insulin therapy specifically. Additionally, this paper identifies areas for future research to define optimal computer-based IIT process execution: the frequency and effect of manual data entry error of blood glucose values, the frequency and effect of nurse overrides of CDSS insulin dosing recommendations, and comprehensive ethnographic study of CDSS for IIT.

Keywords: intensive insulin therapy, clinical protocols, clinical decision support systems, organizational change, organizational structure, hospital information systems, evaluation, sociology of technology

Introduction

The U.S. National Research Council recently endorsed the use of clinical decision support systems (CDSS) and “organizational systems-level research” of health information technology to help drive healthcare transformation [1]. Historically evaluations of CDSS have focused on practitioner performance [2] rather than social, organizational, and contextual factors [3, 4]. Kaplan noted that CDSS evaluation studies measure CDSS effects on clinical performance, use experimental study designs or randomized controlled trials, disregard naturalistic study methods, ignore contextual issues surrounding system usage, investigate the perspectives of physicians rather than other clinical roles, and consider only the CDSS intervention, not other clinical information systems in use [3]. The reporting of findings in the literature reflects a rationalist scientific orientation [3] and shows clinical and medical informatics investigators' preferences toward objectivist rather than subjectivist approaches to evaluation [5]. Although subsequent reviews have identified dimensions of workflow integration as critical to CDSS success [6], researchers have yet to fully embrace the National Research Council's directives or address the gaps identified by Kaplan.

Studies of clinical decision support systems for intensive insulin therapy (IIT), a treatment combining frequent blood glucose monitoring and insulin drip adjustments to maintain tight glucose control [7], follow the general CDSS evaluation trend. Investigations using experimental designs have demonstrated improved clinician protocol adherence and achievement of target glucose levels using computer-based IIT protocols instead of paper-based versions [8-19]. However, these evaluations have paid little attention to the context of interventions, including the complex interaction between staff, testing devices, and computers that may result in inefficiency and error. Nurses use computer-based IIT advisors to document care and calculate insulin doses, but investigations mostly rely on anecdotal feedback to understand nurse perspectives of CDSS and rarely consider CDSS usage with respect to other care processes and clinical information systems. The literature describes paper-based IIT protocol implementation barriers [20] and effects on nurse work [21] but does not explore the complexity and organizational change related to computer-based IIT approaches.

Understanding the mechanisms of effective intensive insulin therapy CDSS is important because IIT is the standard of care for critically ill patients [22]. In 2001 the Leuven study demonstrated morbidity and mortality improvements through an intensive insulin therapy protocol [7], and subsequent studies at other institutions have produced similar results [23, 24]. However, a 2008 meta-analysis of randomized trials raised concerns about the therapy's mortality benefit and safety [25]. Differences in care protocols ranging from nutrition provisions [26] to target blood glucose ranges [26, 27], insulin administration [28], and intended patient populations [29] may explain variation in IIT outcomes, but researchers have not determined comprehensive solutions, especially ones that address computer-based approaches.

Although care protocols define the decision-making behavior clinicians should exhibit under certain conditions [30] and represent the evidence-based, formal structure of healthcare organizations, actual work activities usually differ from official practice definitions [31]. In patients treated with computer-based intensive insulin therapy in the surgical intensive care unit at Vanderbilt University Hospital, researchers found fourteen percent of blood glucose measurements were not taken on time [32]. Significant relationships between late blood glucose measurements and episodes of hyper- and hypoglycemia [32] as well as blood glucose variability and mortality [33] suggest that workflow may be a factor in computer-based IIT performance and patient outcomes [32]. In sociology and organizational studies, institutional theory [31, 34-38] examines the way rules, policies, and procedures affect and are affected by “assumptions, norms, values, choices, and interactions” [36]. This approach has informed investigations of information technology in law [39], banking [40], and research workplaces [41], and informatics researchers have focused on similar issues to influence system design [42-44]. To improve intensive insulin therapy protocol performance and patient outcomes, researchers and practitioners can use institutional theory to address care process execution issues related to human behavior.

This paper takes a subjectivist approach [4] to the study of computer-based intensive insulin therapy and illustrates the need for additional research in two parts: 1) a literature review, which uses institutional theory to take inventory of formal structure and social organization [35] reported in computer-based IIT evaluations, and 2) a case study that builds on the literature review and emphasizes social, organizational, and contextual aspects typically absent from computer-based IIT evaluations. The literature review can potentially serve as a source for other CDSS evaluators interested in social, organizational, and contextual elements, and the case study shares the experience of computer-based IIT at one institution so other institutions can make informed decisions. Overall the analysis shows a gap in the computer-based IIT literature concerning complexity of protocol execution, opportunity for error in staff-device-CDSS interaction, effects on other workflow and care processes, and the magnitude of organizational change necessary for implementation.

Literature Review of Computer-based Intensive Insulin Therapy Evaluations

In May 2008 we searched ISI Web of Science for articles citing the Leuven study (1,783 articles) and containing the keyword “protocol” (129 articles). Because the Leuven study played a significant role in IIT protocols becoming the standard of critical care, we used it to focus our search. From the “protocol” corpus we identified fifteen evaluations of computer-based IIT protocols. Fourteen evaluations used experimental designs or randomized trials, and one was a practice report. The studies examined eighteen intensive care units in twelve healthcare organizations excluding the hundreds of sites evaluated in a longitudinal study of a commercial product [10].

One of the authors (TRC) reviewed the studies through the lens of institutional theory [31, 34-38] to identify aspects of computer-based intensive insulin therapy's formal structure—the prescribed, written policies established to govern and evaluate behavior—and social organization of computing—the interaction of people, process, and technology across different locations and over time [35]. Researchers have used these dimensions to understand the interdependence of technology and human behavior in shaping organizational activity in banking [40], legal [39] and university research settings [41]. For example, through the implementation of a locally hosted digital legal library, a metropolitan court system sought to improve attorneys' legal research and limit cost [39]. Formal structure, manifested in system policies, defined access according to professional role and discouraged use of expensive remote subscription services in favor of the local digital library [39]. The social organization of computing was critical to effective digital library usage: convenient terminal access, workflow integration, favorable attitudes toward computing, separate computer work areas for competitive attorney groups, individualized training, and the emergence of social norms regarding digital library usage in courts [39].

Kling describes three main components of the social organization of computing: equipment configurations, skills and roles, and support infrastructure [35]. Equipment configurations involve the locations of hardware, software, functionality modules within software, and peripherals; skills and roles encompass the various members of an organization who use, supply, or affect an information system's data; and support infrastructure concerns the ways that system stakeholders (e.g. users and managers) obtain assistance and direction [35]. Additionally, temporal aspects of system use, such as periodic (e.g. morning vs. evening) and long-term change over time (e.g. initial vs. established patterns of usage), are salient for analysis [35].

Formal structure of computer-based intensive insulin therapy

Formal structure, defined as the codified procedures intended to govern and evaluate behavior, is well-documented in the computer-based intensive insulin therapy literature. Researchers frequently reported protocol algorithm details as well as evaluation measures in terms of practitioner and patient outcomes, which have been previously defined [2].

Protocol algorithm details

Computer-based IIT approaches used two main algorithmic techniques to determine insulin dosing: linear equations [8-11, 17, 45] based on the work of Bode [46] and White [47] and conditional logic [15, 16, 18, 48]. Other approaches included model predictive control [49] and engineering control math [13]. Most researchers disclosed the logic of computer-based insulin dosing algorithms [8-10, 14-18, 45, 48], and some researchers disclosed previously [8, 9] or concurrently used [13] paper-based IIT protocols.

Practitioner outcomes

Measures of practitioner outcomes included blood glucose target achievement (e.g. time in target range [8, 13-15, 19, 45, 48], percentage in target range [8, 9, 11, 13, 16, 19], time before reaching target range [9-11, 13, 14, 16, 19, 45]), blood glucose mean and median (e.g. overall [9, 11, 13, 15, 17, 18], after 24 hours [14, 19], per day [8, 15, 45]), total blood glucose measures (e.g. overall [9, 13, 15, 16, 18, 45] and per day [8, 14, 15, 19]), hyperglycemia [8, 9, 11, 14-16, 19, 45], hypoglycemia [8-11, 13, 16, 18, 19, 45], insulin administration totals [10, 19], and protocol compliance (e.g. time to initiation of protocol [8, 14, 19], measurement and dosing per protocol schedule [16, 19, 48], administration of recommended insulin dose [19, 48]). Evaluation measures and clinical performance varied between studies. Only one study noted a low percentage of blood glucose results in target range and high percentage of tests not performed on time [16]. Based on rare occurrences of hypoglycemia and reductions in hyperglycemia, most studies deemed IIT protocols “safe and effective” for glucose management.

Patient outcomes

Few studies evaluated patient outcomes in addition to practitioner outcomes [9, 17, 18]. Despite demonstrating improved practitioner outcomes, two studies showed no difference in patient outcomes [9, 18] while another showed reduced morbidity and length of stay but increased mortality [17]. Most studies were preliminary and lacked statistical power to detect patient outcome changes.

Social organization of computer-based intensive insulin therapy

Compared to formal structure, social organization—the interaction of people, process, and technology—was less consistently reported in the computer-based intensive insulin therapy literature. Computer-based approaches to IIT involved various levels of computer systems integration and interaction with testing devices as well as impact on and influence of other care processes and hospital units. The following reviews the social organization of computer-based IIT implementations in terms of equipment configurations, skills and roles, and support infrastructure [35]. Table I provides a summary.

Table I
Social Organization of Computer-based IIT Reported in the Literature

Equipment configurations

Equipment configurations include the placement of computers, software, software functionality, and peripherals within large information systems in particular settings [35]. For computer-based IIT, this includes decision support system location and integration, blood glucose testing device usage, and device-computer interface. Figure 1 depicts the interaction of these elements in computer-based IIT workflow reported in the literature.

Figure 1
Computer-based intensive insulin therapy workflow reported in the literature

IIT CDSS location and integration with clinical data repositories

Clinicians used three IIT CDSS mechanisms: 1) modules embedded within existing primary clinical information systems, including care provider order entry (CPOE) systems, that are accessible from hospital workstations and store blood glucose and insulin dosing data in clinical data repositories (CDR) [8, 9, 11, 14, 16, 48, 49]; 2) “calculators” accessible on a hospital network that do not store data in a CDR [15, 50] and may require additional documentation in a clinical information system [18] and/or use of a preprinted order set [45]; and 3) applications installed on standalone computers1 [14, 49]. Few studies reported location of hospital workstations [48], electronic data interchange with patient monitoring equipment [48], and use of other clinical information systems that are related to or may affect IIT CDSS, workflow, or care processes (e.g. nursing documentation) [18].

Blood glucose testing device usage

Clinicians used handheld glucometers [8, 9, 13, 15-17, 48, 50], non-handheld blood gas analyzers [14, 49], and a combination of both [18] to obtain blood glucose measurements. The importance of handheld glucometers was demonstrated by additional glucometer investment before implementation [18], shortages during implementation [50], and the mechanical failure of a single non-handheld blood gas analyzer temporarily halting protocol use [14].

Interface between CDSS and blood glucose testing devices

Clinicians manually transcribed blood glucose values from testing devices to CDSS [8, 9, 11, 18, 45, 50], automatically transferred test results through docking stations in real time [48], or automatically transferred test results from non-handheld blood gas analyzers in real time [14, 19]. Depending on clinical information systems integration, nurses recorded blood glucose results and insulin rates in both CDSS and nursing documentation tools [18]. Continuous monitoring technology was identified as a possible alternative in the future [10, 16, 49].

Skills and roles

Skills and roles include the various members of an organization who use, supply, or affect an information system's data [35]. For computer-based IIT, this involves nurses as well as care members engaged in other processes that influence system use.

Nurse feedback

Nurse feedback regarding IIT CDSS was mostly positive but evaluation methods lacked rigor. One study used a formal questionnaire to gauge nurse perceptions before and after implementation [14], whereas most studies reported anecdotal nurse feedback related to the interventions' ease of use [8, 18] and increased nursing workload [11, 16, 49]. In one study researchers identified increasing nurse autonomy as a goal of the implementation [50]. Although nurses are the primary users of IIT CDSS, nurse feedback is not a focus of investigations.

Other care processes

Other care processes that may affect IIT and use of IIT CDSS were frequently overlooked. One study reported the concurrent activity of a diabetes disease management service [45]. Demands from surgery and imaging occasionally interrupted IIT usage [18], and ICU nurses administered steroids [9, 18] and nutrition [9, 14, 16, 18] that may have affected patients' blood glucose levels. Description of IIT CDSS workflow integration with respect to the disruptive nature of healthcare was not present in the literature.

Support infrastructure

Support infrastructure concerns the ways in which system stakeholders (e.g. users and managers) obtain assistance and direction [35]. For computer-based IIT, this focuses on the activities of information technology professionals, care team members, and hospital administrators.

Design and training

Multidisciplinary teams consisting of physicians, nurses, pharmacists, and informaticians were responsible for the creation of IIT care protocols and computer-based advisors [9, 16, 18, 50]. Some approaches to computer-based IIT stressed the importance of embedding decision support systems in clinical workflow [8, 9, 14]. Training procedures included pre-implementation multidisciplinary instruction [9, 14] and web-based nurse training [9] as well as “continued need for staff instruction and compliance regarding the protocol” [45].

Diffusion of IIT CDSS

All computer-based IIT protocols originated in an ICU setting, and many diffused to other ICUs within the same institution [9-11, 19, 45] as well as medical-surgical floors [10, 11, 45], recovery [45], labor and delivery [45], and progressive care units [11]. Two approaches diffused to multiple hospitals after initial usage [10, 11], and two research teams created organizations to advance research and adoption of their respective systems [10, 49].

Summary of literature review

Computer-based intensive insulin therapy studies reported formal structure consistently and social organization inconsistently, which reflects the objectivist approach predominating CDSS investigations [3] and the norms of the clinical literature. Most evaluations provided algorithm details and measurements of practitioner performance, but consideration of real world system usage and effects on healthcare organizations in terms of equipment configurations, skills and roles, and support infrastructure varied. Although most interventions relied on the use of handheld glucometers, none recognized the complexity and capacity for error of nurse-device-computer interaction. Studies irregularly described the effect of computer-based IIT on other care processes and clinical information systems usage and vice versa. Although nurses were the primary users of computer-based IIT interventions, most evaluations did not explicitly evaluate nurse feedback. Some studies described the importance of workflow integration and multidisciplinary cooperation, but the literature lacked a comprehensive description of unintended consequences and change management strategies. Evaluations did not address social, organizational, and contextual issues related to computer-based intensive insulin therapy.

Case Study: Intensive Insulin Therapy in the Vanderbilt University Hospital SICU

Intensive insulin therapy represents a set of organizational changes involving the recursive relationship between formal structure and actual work practices, a process which the following case study demonstrates. Based on review of the literature, most computer-based intensive insulin therapy studies ignore the social, organizational, and contextual aspects that explain the effectiveness of interventions. By examining the transition from ad hoc sliding-scale insulin therapy to standardized intensive insulin therapy in the surgical intensive care unit at Vanderbilt University Hospital, this case study illustrates aspects usually omitted from evaluations of computer-based IIT: the importance of local leadership, the expenditure of labor and capital, the relationship between the ICU and other organizational entities, and the influence of technology on clinical process and vice versa. Additionally, the case draws attention to consequences of computer-based IIT—staff- device-computer interaction and the therapy's effect on other care processes—that represent opportunities for error and require additional research. Rather than treat the research setting as static, we aim to show how its dynamic properties change over time and affect and are affected by physicians, nurses, laboratory personnel, and informatics personnel.

We used naturalistic methods [51] to create a three stage chronological narrative of insulin therapy in the study site: glycemic regulation before IIT, paper-based IIT, and computer-based IIT. For stage one, we interviewed nurses, physician leadership, and laboratory personnel. For stage two, we reviewed colleagues' publications [8] and interviewed nurses and informatics support staff. For stage three, we interviewed nurses, physician leadership, informatics support staff, and laboratory personnel in addition to directly observing workflow and reviewing colleagues' publications [8]. Preceding the narrative stages, we also gathered site background information based on review of internal documents and interviews with unit leadership. The Vanderbilt University Institutional Review Board approved this study.

Site background

At Vanderbilt University Hospital (VUH), a large academic urban tertiary care center consisting of 501 beds, the surgical intensive care unit (SICU) admits 1,300 patients each year. The SICU occupies a single floor of the hospital and has a horseshoe layout with a nurse station and supply room in the middle and 21 beds lining the exterior. Each patient room contains at least one clinical workstation connected to the hospital network. Additional workstations are located on mobile carts, at the central nurse's station, adjacent to isolation rooms in antechambers, and throughout the corridors. At VUH the use of electronic patient care information systems has a fifteen year history and is engrained in clinician culture. Clinicians use locally developed electronic medical record and provider order entry systems in addition to vendor applications for ancillary functions and nursing documentation.

Since 2001 the SICU has been under the leadership of a medical director focused on strengthening unit operations as well as promoting collaboration with other hospital units. Efforts include increasing the number of SICU beds from 14 to 21, expanding the use of evidence-based guidelines, shifting cardiovascular surgery patients out of the SICU into a new intensive care unit, collaborating more closely with trauma ICU, creating a full-time SICU critical care service to replace an elective service comprised of critical care and anesthesiology faculty, and facilitating the creation of a separate emergency general surgery service. During this period of growth, SICU experienced increased patient volume and illness severity compared to pre-2001 levels.

Glycemic Regulation Before Intensive Insulin Therapy

Dependence on clinical judgment, inconsistent care processes, and documentation difficulties characterized sliding scale insulin (SSI) therapy, the standard of care for SICU glycemic regulation prior to paper-based intensive insulin therapy. All diabetic patients, as well as non-diabetics with blood glucose issues caused by sepsis or medications, received SSI treatment. Although nurses generally contacted physicians when a blood glucose measurement exceeded 150-200 mg/dL, no explicit criteria defined the threshold of hyperglycemia and when a SICU patient should begin insulin therapy. Physicians' SSI orders defined blood glucose measurement intervals and specific insulin doses for blood glucose ranges. Less experienced nurses adhered to SSI orders whereas more experienced nurses would use clinical judgment (e.g. accounting for a patient's glucose-affecting therapies) in determining subcutaneous insulin injection dosing and subsequent blood glucose monitoring intervals (e.g. Q1H to Q6H) using LifeScan Basic® handheld glucometers. In addition to subcutaneous sliding scale insulin, patients received insulin infusions along with electrolytes as part of total parenteral nutrition. This dosing was also non-standardized and relied on physician discretion. Physicians and nurses depended on experience to initiate therapy, adjust subcutaneous insulin doses, and monitor blood glucose levels.

SSI data management was problematic. Following each blood glucose measurement and insulin administration, nurses documented data on the paper ICU flowsheet, daily glucose log, and medication administration record. Once per day a carbon copy of each patient's daily glucose log was transported to the laboratory for entry into the laboratory information system (LIS). Recording blood glucose (BG) results in the LIS enabled the institution to track resource utilization, assess point-of-care testing compliance, manage billing, and meet regulatory requirements. The LIS also interfaced with the clinical data repository, which clinicians accessed from hospital workstations to view lab results. However, blood glucose and insulin data appeared in the CDR only about 40% of the time: SICU staff were often too busy to transport logs, and the laboratory did not routinely send personnel to SICU to check compliance and collect log sheets. Physicians turned to paper charts instead to obtain blood glucose and insulin data.

A non-protocol-based approach to care, sliding scale insulin permitted variability in clinical decision making. Treatment using SSI was reactive rather than proactive in that it treated hyperglycemia instead of attempting to prevent it, which allowed fluctuation of blood glucose levels and risk of hyper- and hypoglycemia in patients [52]. Non-standardized care and workflow breakdowns typified sliding scale insulin in the SICU.

Paper-based Intensive Insulin Therapy

In August 2003 the VUH SICU implemented a paper-based intensive insulin therapy protocol [8] based on the Leuven study [7], but labor requirements, task complexity, and workflow integration hindered protocol performance. The protocol increased nurse workload by requiring blood glucose measurements, insulin rate adjustments, and subsequent documentation at two hour intervals for both diabetic and non-diabetic patients. Under the new protocol, nurses initiated intensive insulin therapy when a patient's BG level exceeded 110 mg/dL instead of waiting for BG levels to reach a discretionary level as under the previous SSI standard of care. This increased the number of patients treated with insulin. After performing BG tests at the bedside using LifeScan Basic glucometers, nurses consulted the paper medication administration record for the protocol's instructions to manually calculate insulin titrations, a process which required nurse interpretation of the protocol (e.g. “increase infusion by 1-2 units/hr” or “decrease infusion by 25-50%”). Nurses then recorded BG and insulin data on the ICU flowsheet, daily glucose log, and medication administration record. Further complicating implementation was a local nursing shortage. Staff disagreed with the protocol, ignored recommendations, lacked time to perform calculations, and made mental mistakes. Although the purpose of the protocol was to improve care through standardization, variability persisted while demands on nurses increased and potential patient safety threats emerged.

The new protocol also created difficulty for the laboratory, which affected SICU staff. Because of the increase in blood glucose tests performed, laboratory personnel required more time to transcribe test values into the system, which resulted in a processing backlog. Illegible daily glucose logs caused laboratory personnel to occasionally transcribe BG results incorrectly, which caused values in the CDR to not match up with paper documentation. Nurses and physicians became frustrated because they were unable to access accurate BG values through the CDR in a timely fashion.

Overall physicians and nurses were not satisfied with IIT's impact on work processes, and auditing protocol performance was labor intensive due to manual chart review. The average patient blood glucose value, 140-150 mg/dL, exceeded the target of 80-110 mg/dL. Despite organizational changes to standardize patient care, nurse work processes varied and practitioner outcomes did not meet goals.

Computer-based Intensive Insulin Therapy

A multidisciplinary team implemented a computer-based advisor in the VUH SICU that improved intensive insulin therapy performance [8] and produced unintended consequences. In May 2004 an informatics faculty member approached the SICU medical director, a surgeon and critical care physician respected by staff, about developing a computerized IIT approach to improve protocol adherence and capture of process variables for subsequent analysis (e.g. blood glucose values, insulin doses). The SICU medical director commissioned a team of staff nurses, nursing leadership, pharmacists, physicians, and informaticians to assess the IIT process and develop the functionality and interface for a clinical decision support system. Because care provider order entry usage was a regular part of clinical workflow, the team decided to embed the decision support module in the institution's CPOE system. The team tested the intervention and worked with “super user” nurses to refine the tool's ease of use, validate its effectiveness, and assuage concerns about computer-based dosing recommendations. The team also created a training regimen for staff consisting of classroom training for nurses, physician training through orientation, pharmacist training through rounds, continuous informatics staff support, and ad hoc instruction from a SICU nurse practitioner educator.

A separate laboratory investment decision influenced nurse IIT workflow, CDSS design, and project timing. Independent of the SICU in September 2004, the laboratory replaced all glucometers across the institution with Lifescan® SureStep® Pro™ devices ($550 each) and installed a data infrastructure consisting of docking stations ($300 each) and a software interface ($90,000 5-year contract) to automatically transfer blood glucose results from testing devices to the LIS and CDR, thus alleviating the laboratory daily glucose log processing problem. However, test results took up to ten minutes to transfer from device to CDR. Furthermore, devices would not send results until errors were resolved, which occasionally lengthened the transfer process. Data transfer issues coupled with the time-sensitive nature of IIT affected CDSS design: nurses would manually transcribe the latest blood glucose value from the glucometer to the CDSS. To initiate a blood glucose measurement, a nurse used the SureStep® Pro's™ integrated barcode reader to scan barcodes attached to his name badge and the patient's bedside. If barcode scanning failed, a nurse manually entered identification numbers for himself and/or the patient. For legal and billing purposes, the laboratory required BG results entered directly into the LIS by laboratory personnel or automatic device transfer, not manual nurse transcription. Once per shift nursing assistants collected devices and placed them in one of two docking stations to transfer test results and accompanying identification information. After use of SureStep® glucometers became a regular part of workflow, the SICU team resumed its CDSS implementation effort in December 2004.

The computer-based IIT approach introduced a tool to assist nurses with glucose maintenance as well as a practice change to increase physician involvement in glycemic regulation. Instead of starting the protocol when a blood glucose reading exceeded 110 mg/dL, a nurse contacted a physician to initiate therapy using the CDSS module, which consisted of two parts [8]: an initiation screen for a physician to specify care and “notify house officer” parameters, and an insulin rate adjustment screen for a nurse to manually enter blood glucose values (Figure 2). Following a physician's one-time use of the initiation screen, the nurse accessed the CDSS module according to the protocol schedule (usually Q2H) in order to document blood glucose results and calculate new insulin titrations. The CDSS module utilized a linear equation to determine an insulin titration [8], which eliminated the need for nurses to manually calculate insulin infusion rates. However, nurses could override the CDSS module's recommendations and enter an insulin titration using their clinical judgment when necessary (e.g. simultaneous administration of glucose-affecting medication). After using the CDSS, nurses manually adjusted rates of pharmacy-prepared regular insulin drips (150 units in 150mL normal saline solution with a 24 hour expiration) on Alaris® infusion pumps equipped with Guardrails® software, which was not configured to transfer infusion data to the CDR. Although several years later the institution implemented a barcode medication administration system integrated with other clinical software for administering intermittent medications, nurses did not use it for infusions.

Figure 2
Screenshots of clinical decision support system

Compared to its paper predecessor, the SICU computer-based IIT protocol increased protocol adherence, reduced time to initiate treatment, expanded the percentage of blood glucose readings in the target range, and simplified record keeping [8]. The hospital's medical, neurological, cardiovascular, and trauma ICUs adopted the same computer-based approach to IIT, and the trauma ICU demonstrated glycemic regulation improvements using the intervention [9]. Neither the SICU or trauma ICU studies were sufficiently powered to detect patient outcome improvements, but in terms of glycemic regulation improvements, the computer-based approach to IIT was a success at the institution.

In addition to improving practitioner performance, the intervention produced unintended consequences related to workflow and technology. First, IIT and other redesigned clinical activities contributed to increased overall CPOE usage, which resulted in clinicians waiting to use terminals in SICU. In response, the institution purchased additional workstations. Second, the clinical data repository's blood glucose and insulin data appeared in duplicate—one set of values entered manually by nurses into the CDSS module, the other captured from the glucometer—with slightly different timestamps and occasionally different values. This resulted in visual clutter in CDR data displays, which may have contributed to clinician confusion or cognitive overload. Third, nurses “double documented” blood glucose and insulin values in the CDSS module and an electronic nursing documentation system, which was implemented two years after the introduction of computer-based IIT and the completion of protocol evaluations in the SICU [8] and trauma ICU [9]. This resulted in a third set of values appearing in the clinical data repository. Furthermore, the approach to computer-based IIT assumed nurses never made errors when transcribing blood glucose values to calculate and adjust insulin doses. Despite these issues, computer-based IIT remains the standard of care for critically ill patients at VUH.

Summary of case study

In the transition from sliding scale insulin to paper-based IIT to computer-based IIT, Vanderbilt University Hospital enacted considerable organizational changes related to evidence-based protocol development, nurse workload, physician involvement, blood glucose testing and infrastructure, and informatics development and support. Forces beyond SICU control—a local nursing shortage, the laboratory's decision to upgrade glucometers, and the institution's decision to implement nursing documentation software—affected the trajectory of intensive insulin therapy efforts over time, but SICU leadership and multidisciplinary cooperation helped ensure the project's success. Other institutions may experience similar organizational changes as part of their computer-based IIT efforts. Changes to glycemic regulation, glucometer usage, and computerization occurred gradually over time at VUH, which conceivably enabled stakeholders to adapt to process modifications more easily. In contrast, other institutions may face greater change management challenges if abruptly shifting from sliding scale insulin to computer-based IIT. Computer-based intensive insulin therapy is a complex, multifaceted organizational undertaking that requires substantial commitment to change and presents opportunities for further inefficiency and error reduction.

Discussion

Kaplan's themes of clinical decision support system evaluation [3] are present in evaluations of computer-based intensive insulin therapy. In order to optimize computer-based intensive insulin therapy process execution, researchers and practitioners should address social, organizational, and contextual issues determining how and why implementations are successful. From our literature review and case study, three aspects of computer-based IIT appear particularly salient: (1) the relationship between clinical information systems, CDSS, testing devices, users, and error; (2) nurse perspectives; and (3) organizational change.

Technology, users, and error

The interaction of hardware, clinical information systems, clinical decision support modules, blood glucose devices, and clinicians is complex, time consuming, and susceptible to error, yet most evaluations of computer-based IIT take it for granted. For example, a study of computer-based IIT conducted at Vanderbilt University Hospital stated that “[blood glucose] values are downloaded directly from the glucometer to the computer order entry system” [32], which misrepresents the reality of manual data entry and possibility for error inherent in the process. Installation of additional docking stations at each bedside to facilitate data transfer may be cost prohibitive or hindered by slow data transfer times. The purchase of glucometers that transmit data wirelessly across a hospital network to clinical data repositories in a reliable fashion may also be cost prohibitive. Furthermore, controversy surrounds the use of handheld glucometers for intensive insulin therapy due to possible inaccurate results [53-56]. Some studies suggest continuous glucose monitoring technologies can replace handheld glucometers today [57, 58] while others propose additional refinement [59-61] or recommend against their usage [62]. In contrast to computer-based IIT, computer-based anticoagulation therapy [63] relies on a central laboratory's activated partial thromboplastin time results, which are processed less frequently [64] and are arguably more accurate than handheld glucometer test results. For computer-based intensive insulin therapy, the optimal configuration of testing devices, computers, decision support interfaces, and personnel is not yet understood.

Nurse perspectives

Few studies have focused on nurse perspectives regarding intensive insulin therapy, particularly for computer-based approaches, and additional study can potentially improve protocols and workflow. A direct observation study of a paper-based IIT protocol showed that nurses required between three and nine minutes (mean 4.72, SD 1.13, median 4.67) to obtain a testing device, measure blood glucose, and adjust insulin [21]. A separate time-motion study found nurses required 20-30 minutes to complete IIT tasks and document care [65]. Times varied due to treatment differences for hypoglycemia, hyperglycemia, and euglycemia [65] as well as nurses locating devices, troubleshooting devices, caring for patients with isolation precautions, and occasionally ignoring hygiene and safety requirements [21]. Such issues may also influence provision of computer-based IIT, and CDSS and other computer system usage during IIT administration may have other unintended consequences that add to nurse work or detract from patient safety. In a study of computer-based IIT2, nurses indicated the following reasons for declining CDSS recommendations: patient blood glucose trends, concurrent administration of medications prepared in a glucose solution, nutrition changes, concurrent epinephrine administration, hypothermia, agitation, and previously entered incorrect data [66]. The results of this study show some of the effects of CDSS on IIT and demonstrate the value of nurse-focused evaluation of computer-based IIT in making workflow and care barriers explicitly understood. New dosing algorithms can potentially incorporate such factors so that IIT protocols reflect the realities of clinical practice and judgment.

Organizational change

The same computer-based intensive insulin therapy protocol used in two hospitals, or two units in the same hospital, might produce variability in social processes and clinical performance. Examining the social organization of computer-based IIT evaluations shows that the effects of computer-based IIT implementations on healthcare organizations are not explicitly reported. Our case study demonstrates how a surgical intensive care unit with strong leadership and institutional informatics support overcame technological and organizational barriers to implement computer-based IIT. Although other ICUs in the institution now use the same computer-based IIT approach, the intervention may or may not appropriately match workflow, organizational, and clinical needs because it was designed for the SICU. A recent multi-site IIT trial [67], which showed increased mortality for patients treated with IIT versus those treated with conventional therapy, used the same computer-based IIT dosing calculator in all sites [68]. The researchers did not explore computer-based IIT process execution across sites although such issues may have affected clinical performance.

Future research

To understand computer-based IIT usage, future studies should combine quantitative and qualitative methods. First, by comparing manually entered blood glucose data and automatically captured values from glucometers, we can determine the frequency and effect of incorrect data on insulin dosing and blood glucose variability. Most computer-based IIT studies utilize handheld glucometers and assume the transcription of blood glucose values from testing devices to CDSS is error free. A study examining ventilator settings automatically captured from a device versus manually entered into a computerized CDSS showed 3.9% of computer-generated recommendations contained incorrectly entered data [69]. Blood glucose value errors may potentially contribute to blood glucose variability, which has been associated with mortality [33].

Second, investigating the impact of CDSS insulin dose overrides can assess the effectiveness of nurses' clinical judgment. Studies of medication-related CDSS embedded in CPOE systems show physician override rates of computer-based recommendations as high as 91% [70, 71], and researchers suggest using quantitative and qualitative methods to understand clinician-CDSS interaction [70]. Nurses deviate from CDSS suggestions when a clinical situation is more complex than a computer algorithm's parameters [66]. However, little is known about whether nurses' clinical judgment is appropriate under these circumstances. Quantifying the frequency and effect of insulin dose overrides on blood glucose variability can potentially answer this question. Examining medication administration records and clinical documentation corresponding to nurse overrides of CDSS recommendations may provide indication of additional variables for IIT dosing algorithms to consider (e.g. corticosteroids).

Third, the use of ethnographic methods to study computer-based intensive insulin therapy can potentially lead to software and process enhancements. An approach from anthropology, ethnography has been used in clinical research to improve surgical resident handoffs [72] and in informatics research to identify and resolve incorrect software design assumptions [42, 73]. Extensive direct observation of clinicians using CDSS for IIT in the field can reveal benefits and drawbacks of the current approach with respect to computer system usage, care processes, and issues currently unknown. Additionally, ethnographic study of computer-based IIT in multiple ICUs can potentially show site-specific differences in social organization of the intervention that may affect clinical performance. By understanding the use of computer-based IIT in real world settings, researchers and practitioners can make care workflow and protocol modifications to potentially achieve the morbidity and mortality improvements demonstrated in the Leuven study.

Limitations

There are limitations to this study. First, we examined intensive insulin therapy in one intensive care unit at a single institution with sophisticated clinical informatics systems. Findings may not generalize to sites with less informatics development. Future research will examine additional ICUs at Vanderbilt University Hospital and then proceed to additional institutions. Second, other theoretical approaches might be more illuminative than institutional theory's social organization of computing in examining computer-based intensive insulin theory. For example, social interactionism, “fit,” and “the 4 C's” may be potentially useful methods [4].

Conclusion

Our analysis contributes to the understanding of computer-based intensive insulin therapy's social, organizational, and contextual aspects. More broadly, this paper addresses the underreported elements explaining how and why clinicians use CDSS interventions. We suggest future IIT CDSS research involve quantifying error, assessing clinical judgment in overriding CDSS recommendations, and directly observing nurse use of IIT CDSS with respect to other care processes and clinical information systems. Researchers and practitioners can use this study to approach computer-based intensive insulin therapy and clinical decision support system improvement projects.

Summary Table

What was already known:

  • Evaluations of clinical decision support systems (CDSS) generally ignore social, organizational, and contextual factors explaining their effectiveness or ineffectiveness
  • CDSS for intensive insulin therapy have improved protocol adherence and performance, but controversy surrounds the treatment's mortality benefit and safety

What this study added to our knowledge:

  • Computer-based intensive insulin therapy requires substantial organizational change and introduces additional complexity with unintended consequences including error
  • Examining informatics literature and issues through the lens of institutional theory may assist researchers and practitioners identify social, organizational, and contextual aspects and solve problems

Acknowledgements

Mr. Campion received support from National Library of Medicine Training Grant NLM T15 007450-07. The authors thank Gwen Holder, RN, MSN for internal document assistance.

Footnotes

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

Conflict of Interest Statement The authors have no conflicts of interest to report.

1In these studies the authors described use of the CDSS in embedded and standalone configurations.

2This study did not meet literature review criteria as it was published in June 2008 (and not yet indexed in ISI Web of Science) and did not cite the Leuven study.

References

1. Stead W, Lin H. Computational technology for effective health care : immediate steps and strategic directions. National Academies Press; Washington, DC: 2009. [PubMed]
2. Garg AX, Adhikari NK, McDonald H, Rosas-Arellano MP, Devereaux PJ, Beyene J, et al. Effects of computerized clinical decision support systems on practitioner performance and patient outcomes: a systematic review. Jama. 2005 Mar 9;293(10):1223–38. [PubMed]
3. Kaplan B. Evaluating informatics applications--clinical decision support systems literature review. International journal of medical informatics. 2001 Nov;64(1):15–37. [PubMed]
4. Kaplan B. Evaluating informatics applications--some alternative approaches: theory, social interactionism, and call for methodological pluralism. International journal of medical informatics. 2001 Nov;64(1):39–56. [PubMed]
5. Friedman CP, Wyatt JC. Evaluation Methods in Biomedical Informatics (Health Informatics) Springer-Verlag New York, Inc.; 2005.
6. Kawamoto K, Houlihan CA, Balas EA, Lobach DF. Clinical research, editor. Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success. BMJ. 2005 Apr 2;330(7494):765. [PMC free article] [PubMed]
7. van den Berghe G, Wouters P, Weekers F, Verwaest C, Bruyninckx F, Schetz M, et al. Intensive insulin therapy in the critically ill patients. The New England journal of medicine. 2001 Nov 8;345(19):1359–67. [PubMed]
8. Boord JB, Sharifi M, Greevy RA, Griffin MR, Lee VK, Webb TA, et al. Computer-based insulin infusion protocol improves glycemia control over manual protocol. J Am Med Inform Assoc. 2007 May-Jun;14(3):278–87. [PMC free article] [PubMed]
9. Dortch MJ, Mowery NT, Ozdas A, Dossett L, Cao H, Collier B, et al. A computerized insulin infusion titration protocol improves glucose control with less hypoglycemia compared to a manual titration protocol in a trauma intensive care unit. Jpen. 2008 Jan-Feb;32(1):18–27. [PubMed]
10. Davidson PC, Steed RD, Bode BW. Glucommander: a computer-directed intravenous insulin system shown to be safe, simple, and effective in 120,618 h of operation. Diabetes care. 2005 Oct;28(10):2418–23. [PubMed]
11. Juneja R, Roudebush C, Kumar N, Macy A, Golas A, Wall D, et al. Utilization of a computerized intravenous insulin infusion program to control blood glucose in the intensive care unit. Diabetes technology & therapeutics. 2007 Jun;9(3):232–40. [PubMed]
12. Chase JG, Shaw GM. Is there more to glycaemic control than meets the eye? Critical care (London, England) 2007;11(4):160. [PMC free article] [PubMed]
13. Saager L, Collins GL, Burnside B, Tymkew H, Zhang L, Jacobsohn E, et al. A randomized study in diabetic patients undergoing cardiac surgery comparing computer-guided glucose management with a standard sliding scale protocol. J Cardiothorac Vasc Anesth. 2008 Jun;22(3):377–82. [PubMed]
14. Vogelzang M, Zijlstra F, Nijsten MW. Design and implementation of GRIP: a computerized glucose control system at a surgical intensive care unit. BMC medical informatics and decision making. 2005;5:38. [PMC free article] [PubMed]
15. Meynaar IA, Dawson L, Tangkau PL, Salm EF, Rijks L. Introduction and evaluation of a computerised insulin protocol. Intensive care medicine. 2007 Apr;33(4):591–6. [PubMed]
16. Shulman R, Finney SJ, O'Sullivan C, Glynne PA, Greene R. Tight glycaemic control: a prospective observational study of a computerised decision-supported intensive insulin therapy protocol. Critical care (London, England) 2007;11(4):R75. [PMC free article] [PubMed]
17. Toschlog EA, Newton C, Allen N, Newell MA, Goettler CE, Schenarts PJ, et al. Morbidity reduction in critically ill trauma patients through use of a computerized insulin infusion protocol: a preliminary study. The Journal of trauma. 2007 Jun;62(6):1370–5. discussion 5-6. [PubMed]
18. Thomas AN, Marchant AE, Ogden MC, Collin S. Implementation of a tight glycaemic control protocol using a web-based insulin dose calculator. Anaesthesia. 2005 Nov;60(11):1093–100. [PubMed]
19. Vogelzang M, Loef BG, Regtien JG, van der Horst IC, van Assen H, Zijlstra F, et al. Computer-assisted glucose control in critically ill patients. Intensive care medicine. 2008 Apr 4; [PMC free article] [PubMed]
20. Anger KE, Szumita PM. Barriers to glucose control in the intensive care unit. Pharmacotherapy. 2006 Feb;26(2):214–28. [PubMed]
21. Aragon D. Evaluation of nursing work effort and perceptions about blood glucose testing in tight glycemic control. Am J Crit Care. 2006 Jul;15(4):370–7. [PubMed]
22. Dellinger RP, Carlet JM, Masur H, Gerlach H, Calandra T, Cohen J, et al. Surviving Sepsis Campaign guidelines for management of severe sepsis and septic shock. Critical care medicine. 2004 Mar;32(3):858–73. [PubMed]
23. Grey NJ, Perdrizet GA. Reduction of nosocomial infections in the surgical intensive-care unit by strict glycemic control. Endocr Pract. 2004 Mar-Apr;10(Suppl 2):46–52. [PubMed]
24. Krinsley JS. Effect of an intensive glucose management protocol on the mortality of critically ill adult patients. Mayo Clinic proceedings. 2004 Aug;79(8):992–1000. [PubMed]
25. Soylemez Wiener R, Wiener DC, Larson RJ. Benefits and risks of tight glucose control in critically ill adults: a meta-analysis. Jama. 2008 Aug 27;300(8):933–44. [PubMed]
26. Boulkina LS, Braithwaite SS. Practical aspects of intensive insulinization in the intensive care unit. Current opinion in clinical nutrition and metabolic care. 2007 Mar;10(2):197–205. [PubMed]
27. Wilson M, Weinreb J, Hoo GW. Intensive insulin therapy in critical care: a review of 12 protocols. Diabetes care. 2007 Apr;30(4):1005–11. [PubMed]
28. Finney SJ, Zekveld C, Elia A, Evans TW. Glucose control and mortality in critically ill patients. Jama. 2003 Oct 15;290(15):2041–7. [PubMed]
29. Van den Berghe G, Wilmer A, Hermans G, Meersseman W, Wouters PJ, Milants I, et al. Intensive insulin therapy in the medical ICU. The New England journal of medicine. 2006 Feb 2;354(5):449–61. [PubMed]
30. Morris AH. Developing and implementing computerized protocols for standardization of clinical decisions. Annals of internal medicine. 2000 Mar 7;132(5):373–83. [PubMed]
31. Meyer JW, Rowan B. Institutionalized Organizations: Formal Structure as Myth and Ceremony. The University of Chicago Press. 1977:340–63.
32. Dossett LA, Collier B, Donahue R, Mowery NT, Dortch MJ, Guillamondegui O, et al. Intensive insulin therapy in practice: can we do it? Jpen. 2008 Jan-Feb;33(1):14–20. [PubMed]
33. Dossett LA, Cao H, Mowery NT, Dortch MJ, Morris JM, Jr, May AK. Blood glucose variability is associated with mortality in the surgical ICU. The American surgeon. 2008 Aug;74(8):679–85. discussion 85. [PubMed]
34. Barley SR, Tolbert PS. Institutionalization and structuration: studying the links between action and institution.(Special Issue on Action, Structure and Organizations) Organization Studies. 1997;v18(n1):p93(25).
35. Kling R. The Institutional Character of Computerized Information Systems. Information technology & people. 1989;5(1):7–28.
36. Orlikowski WJ, Barley SR. Technology and Institutions: What Can Research on Information Technology and Research on Organizations Learn from Each Other? Management Information Systems Research Center, University of Minnesota; 2001. pp. 145–65.
37. Tolbert PS, Zucker LG. Institutional Sources of Change in the Formal Structure of Organizations: The Diffusion of Civil Service Reform, 1880-1935. Johnson Graduate School of Management, Cornell University; 1983. pp. 22–39.
38. Zucker LG. Institutional Theories of Organization. Annual Review of Sociology. 1987;13:443–64.
39. Elliott M, Kling R. Organizational usability of digital libraries: Case study of legal research in civil and criminal courts. Journal of the American Society for Information Science. 1997;48(11):1023–35.
40. Barrett M, Walsham G. Electronic Trading and Work Transformation in the London Insurance Market. Information Systems Research. 1999;10(1):1–22.
41. Kling R. Behind the terminal: The critical role of computing infrastructure in effective information systems' development and use. In: Cotterman W, Senn J, editors. Challenges and strategies for research in systems development. John Wiley; New York: 1992. pp. 153–201.
42. Forsythe DE. New bottles, old wine: hidden cultural assumptions in a computerized explanation system for migraine sufferers. Medical anthropology quarterly. 1996 Dec;10(4):551–74. [PubMed]
43. Manning B, Gadd CS. Introducing handheld computing into a residency program: preliminary results from qualitative and quantitative inquiry. Proceedings / AMIA Annual Symposium. 2001:428–32. [PMC free article] [PubMed]
44. Kaplan B, Morelli R, Goethe J. Preliminary Findings from an Evaluation of the Acceptability of an Expert System. Proc AMIA Annu Fall Symp. 1997:865. 1997 1997.
45. Hermayer KL, Neal DE, Hushion TV, Irving MG, Arnold PC, Kozlowski L, et al. Outcomes of a cardiothoracic intensive care web-based online intravenous insulin infusion calculator study at a Medical University Hospital. Diabetes technology & therapeutics. 2007 Dec;9(6):523–34. [PubMed]
46. Bode BW, Braithwaite SS, Steed RD, Davidson PC. Intravenous insulin infusion therapy: indications, methods, and transition to subcutaneous insulin therapy. Endocr Pract. 2004 Mar-Apr;10(Suppl 2):71–80. [PubMed]
47. White NH, Skor D, Santiago JV. Practical closed-loop insulin delivery. A system for the maintenance of overnight euglycemia and the calculation of basal insulin requirements in insulin-dependent diabetics. Annals of internal medicine. 1982 Aug;97(2):210–3. [PubMed]
48. Rood E, Bosman RJ, van der Spoel JI, Taylor P, Zandstra DF. Use of a computerized guideline for glucose regulation in the intensive care unit improved both guideline adherence and glucose regulation. J Am Med Inform Assoc. 2005 Mar-Apr;12(2):172–80. [PMC free article] [PubMed]
49. Plank J, Blaha J, Cordingley J, Wilinska ME, Chassin LJ, Morgan C, et al. Multicentric, randomized, controlled trial to evaluate blood glucose control by the model predictive control algorithm versus routine glucose management protocols in intensive care unit patients. Diabetes care. 2006 Feb;29(2):271–6. [PubMed]
50. Rea RS, Donihi AC, Bobeck M, Herout P, McKaveney TP, Kane-Gill SL, et al. Implementing an intravenous insulin infusion protocol in the intensive care unit. Am J Health Syst Pharm. 2007 Feb 15;64(4):385–95. [PubMed]
51. Erlandson DA. Doing naturalistic inquiry : a guide to methods. Sage; Newbury Park, Calif.: 1993.
52. Katz CM. How efficient is sliding-scale insulin therapy? Problems with a ‘cookbook’ approach in hospitalized patients. Postgraduate medicine. 1991 Apr;89(5):46–8. 51-4, 7. [PubMed]
53. Baig A, Siddiqui I, Jabbar A, Azam SI, Sabir S, Alam S, et al. Comparision between bed side testing of blood glucose by glucometer vs centralized testing in a tertiary care hospital. J Ayub Med Coll Abbottabad. 2007 Jul-Sep;19(3):25–9. [PubMed]
54. Khan AI, Vasquez Y, Gray J, Wians FH, Jr., Kroll MH. The variability of results between point-of-care testing glucose meters and the central laboratory analyzer. Archives of pathology & laboratory medicine. 2006 Oct;130(10):1527–32. [PubMed]
55. Kanji S, Buffie J, Hutton B, Bunting PS, Singh A, McDonald K, et al. Reliability of point-of-care testing for glucose measurement in critically ill adults. Critical care medicine. 2005 Dec;33(12):2778–85. [PubMed]
56. Louie RF, Tang Z, Sutton Dv, Lee JH, Kost GJ. Point-of-care glucose testing: effects of critical care variables, influence of reference instruments, and a modular glucose meter design. Archives of pathology & laboratory medicine. 2000 Feb;124(2):257–66. [PubMed]
57. De Block C, Manuel YKB, Van Gaal L, Rogiers P. Intensive insulin therapy in the intensive care unit: assessment by continuous glucose monitoring. Diabetes care. 2006 Aug;29(8):1750–6. [PubMed]
58. Yamashita K, Okabayashi T, Yokoyama T, Yatabe T, Maeda H, Manabe M, et al. Accuracy and reliability of continuous blood glucose monitor in post-surgical patients. Acta anaesthesiologica Scandinavica. 2009 Jan;53(1):66–71. [PubMed]
59. Corstjens AM, Ligtenberg JJ, van der Horst IC, Spanjersberg R, Lind JS, Tulleken JE, et al. Accuracy and feasibility of point-of-care and continuous blood glucose analysis in critically ill ICU patients. Critical care (London, England) 2006;10(5):R135. [PMC free article] [PubMed]
60. Goldberg PA, Siegel MD, Russell RR, Sherwin RS, Halickman JI, Cooper DA, et al. Experience with the continuous glucose monitoring system in a medical intensive care unit. Diabetes technology & therapeutics. 2004 Jun;6(3):339–47. [PubMed]
61. Logtenberg SJ, Kleefstra N, Snellen FT, Groenier KH, Slingerland RJ, Nierich AP, et al. Pre- and postoperative accuracy and safety of a real-time continuous glucose monitoring system in cardiac surgical patients: a randomized pilot study. Diabetes technology & therapeutics. 2009 Jan;11(1):31–7. [PubMed]
62. Price GC, Stevenson K, Walsh TS. Evaluation of a continuous glucose monitor in an unselected general intensive care population. Crit Care Resusc. 2008 Sep;10(3):209–16. [PubMed]
63. Starmer JM, Talbert DA, Miller RA. Experience using a programmable rules engine to implement a complex medical protocol during order entry. Proceedings / AMIA Annual Symposium. 2000:829–32. [PMC free article] [PubMed]
64. Hylek EM, Regan S, Henault LE, Gardner M, Chan AT, Singer DE, et al. Challenges to the effective use of unfractionated heparin in the hospitalized management of acute thrombosis. Archives of internal medicine. 2003 Mar 10;163(5):621–7. [PubMed]
65. Malesker MA, Foral PA, McPhillips AC, Christensen KJ, Chang JA, Hilleman DE. An efficiency evaluation of protocols for tight glycemic control in intensive care units. Am J Crit Care. 2007 Nov;16(6):589–98. [PubMed]
66. Sward K, Orme J, Jr., Sorenson D, Baumann L, Morris AH. Reasons for declining computerized insulin protocol recommendations: application of a framework. Journal of biomedical informatics. 2008 Jun;41(3):488–97. [PMC free article] [PubMed]
67. Finfer S, Chittock DR, Su SY, Blair D, Foster D, Dhingra V, et al. Intensive versus conventional glucose control in critically ill patients. The New England journal of medicine. 2009 Mar 26;360(13):1283–97. [PubMed]
68. The Normoglycemia in Intensive Care Evaluation (NICE) (ISRCTN04968275) and Survival Using Glucose Algorithm Regulation (SUGAR) Study: development, design and conduct of an international multi-center, open label, randomized controlled trial of two target ranges for glycemic control in intensive care unit patients. American journal of respiratory and critical care medicine 2005. [cited March 27, 2009]; Online supplement:[Available from: http://ajrccm.atsjournals.org/cgi/data/172/11/1358/DC1/1.
69. Vawdrey DK, Gardner RM, Evans RS, Orme JF, Jr., Clemmer TP, Greenway L, et al. Assessing data quality in manual entry of ventilator settings. J Am Med Inform Assoc. 2007 May-Jun;14(3):295–303. [PMC free article] [PubMed]
70. Lin CP, Payne TH, Nichol WP, Hoey PJ, Anderson CL, Gennari JH. Evaluating Clinical Decision Support Systems: Monitoring CPOE Order Check Override Rates in the Department of Veterans Affairs' Computerized Patient Record System. J Am Med Inform Assoc. 2008 Sep-Oct;15(5):620–6. [PMC free article] [PubMed]
71. Weingart SN, Toth M, Sands DZ, Aronson MD, Davis RB, Phillips RS. Physicians' decisions to override computerized drug alerts in primary care. Archives of internal medicine. 2003 Nov 24;163(21):2625–31. [PubMed]
72. Kellogg KC, Breen E, Ferzoco SJ, Zinner MJ, Ashley SW. Resistance to change in surgical residency: an ethnographic study of work hours reform. Journal of the American College of Surgeons. 2006 Apr;202(4):630–6. [PubMed]
73. Forsythe DE. “It's Just a Matter of Common Sense”: Ethnography as Invisible Work. Vol. 8. Computer Supported Cooperative Work (CSCW); 1999. pp. 127–45.