Search tips
Search criteria

Results 1-25 (109)

Clipboard (0)

Select a Filter Below

Year of Publication
more »
1.  Rights and responsibilities of users of electronic health records 
PMCID: PMC3447016  PMID: 22331971
2.  The state of the art in clinical knowledge management: An inventory of tools and techniques 
To explore the need for, and use of, high-quality, collaborative, clinical knowledge management (CKM) tools and techniques to manage clinical decision support content.
In order to better understand the current state of the art in CKM, we developed a survey of potential CKM tools and techniques. We conducted an exploratory study by querying a convenience sample of respondents about their use of specific practices in CKM.
The following tools and techniques should be priorities in organizations interested in developing successful computer-based provider order entry (CPOE) and clinical decision support (CDS) implementations: 1) A multidisciplinary team responsible for creating and maintaining the clinical content; 2) An external organizational repository of clinical content with web-based viewer that allows anyone in the organization to review it; 3) An online, collaborative, interactive, internet-based tool to facilitate content development; 4) An enterprise-wide tool to maintain the controlled clinical terminology concepts. Even organizations that have been successfully using Computer-based Provider Order Entry with advanced Clinical Decision Support features for well over 15 years are not using all of the CKM tools or practices that we identified.
If we are to further stimulate progress in the area of clinical decision support, we must continue to develop and refine our understanding and use of advanced CKM capabilities.
PMCID: PMC2895508  PMID: 19828364
3.  A survey of informatics platforms that enable distributed comparative effectiveness research using multi-institutional heterogeneous clinical data 
Medical care  2012;50(Suppl):S49-S59.
Comparative Effectiveness Research (CER) has the potential to transform the current healthcare delivery system by identifying the most effective medical and surgical treatments, diagnostic tests, disease prevention methods and ways to deliver care for specific clinical conditions. To be successful, such research requires the identification, capture, aggregation, integration, and analysis of disparate data sources held by different institutions with diverse representations of the relevant clinical events. In an effort to address these diverse demands, there have been multiple new designs and implementations of informatics platforms that provide access to electronic clinical data and the governance infrastructure required for inter-institutional CER. The goal of this manuscript is to help investigators understand why these informatics platforms are required and to compare and contrast six, large-scale, recently funded, CER-focused informatics platform development efforts. We utilized an 8-dimension, socio-technical model of health information technology use to help guide our work. We identified six generic steps that are necessary in any distributed, multi-institutional CER project: data identification, extraction, modeling, aggregation, analysis, and dissemination. We expect that over the next several years these projects will provide answers to many important, and heretofore unanswerable, clinical research questions.
PMCID: PMC3415281  PMID: 22692259
Methods; Comparative Effectiveness Research; Organization and Administration; Medical Informatics; Methods
4.  Defining Health Information Technology-related Errors: New Developments Since To Err is Human* 
Archives of internal medicine  2011;171(14):1281-1284.
Despite its promise, recent literature has revealed possible safety hazards of health information technology (HIT) use. The Office of the National Coordinator for HIT recently sponsored an Institute of Medicine committee to synthesize evidence and experience from the field on how HIT affects patient safety. To lay the groundwork for defining, measuring, and analyzing HIT-related safety hazards, we propose that Health information technology-related error occurs anytime HIT is unavailable for use, malfunctions during use, is used incorrectly by someone, or when HIT interacts with another system component incorrectly, resulting in data being lost or incorrectly entered, displayed, or transmitted. These errors, or the decisions that result from them, significantly increase the risk of adverse events and patient harm. In this paper, we describe how a socio-technical approach can be used to understand the complex origins of HIT errors, which may have roots in rapidly evolving technological, professional, organizational, and policy initiatives.
PMCID: PMC3677061  PMID: 21788544
Electronic Health Records; Health Information Technology; Patient Safety; Errors
5.  Legal, Ethical, and Financial Dilemmas in Electronic Health Record Adoption and Use 
Pediatrics  2011;127(4):e1042-e1047.
Electronic health records (EHRs) facilitate several innovations capable of reforming health care. Despite their promise, many currently unanswered legal, ethical, and financial questions threaten the widespread adoption and use of EHRs. Key legal dilemmas that must be addressed in the near-term pertain to the extent of clinicians' responsibilities for reviewing the entire computer-accessible clinical synopsis from multiple clinicians and institutions, the liabilities posed by overriding clinical decision support warnings and alerts, and mechanisms for clinicians to publically report potential EHR safety issues. Ethical dilemmas that need additional discussion relate to opt-out provisions that exclude patients from electronic record storage, sale of deidentified patient data by EHR vendors, adolescent control of access to their data, and use of electronic data repositories to redesign the nation's health care delivery and payment mechanisms on the basis of statistical analyses. Finally, one overwhelming financial question is who should pay for EHR implementation because most users and current owners of these systems will not receive the majority of benefits. The authors recommend that key stakeholders begin discussing these issues in a national forum. These actions can help identify and prioritize solutions to the key legal, ethical, and financial dilemmas discussed, so that widespread, safe, effective, interoperable EHRs can help transform health care.
PMCID: PMC3065078  PMID: 21422090
electronic health records; ethics; medical; confidentiality
6.  A benchmark comparison of deterministic and probabilistic methods for defining manual review datasets in duplicate records reconciliation 
Clinical databases require accurate entity resolution (ER). One approach is to use algorithms that assign questionable cases to manual review. Few studies have compared the performance of common algorithms for such a task. Furthermore, previous work has been limited by a lack of objective methods for setting algorithm parameters. We compared the performance of common ER algorithms: using algorithmic optimization, rather than manual parameter tuning, and on two-threshold classification (match/manual review/non-match) as well as single-threshold (match/non-match).
We manually reviewed 20 000 randomly selected, potential duplicate record-pairs to identify matches (10 000 training set, 10 000 test set). We evaluated the probabilistic expectation maximization, simple deterministic and fuzzy inference engine (FIE) algorithms. We used particle swarm to optimize algorithm parameters for a single and for two thresholds. We ran 10 iterations of optimization using the training set and report averaged performance against the test set.
The overall estimated duplicate rate was 6%. FIE and simple deterministic algorithms allowed a lower manual review set compared to the probabilistic method (FIE 1.9%, simple deterministic 2.5%, probabilistic 3.6%; p<0.001). For a single threshold, the simple deterministic algorithm performed better than the probabilistic method (positive predictive value 0.956 vs 0.887, sensitivity 0.985 vs 0.887, p<0.001). ER with FIE classifies 98.1% of record-pairs correctly (1/10 000 error rate), assigning the remainder to manual review.
Optimized deterministic algorithms outperform the probabilistic method. There is a strong case for considering optimized deterministic methods for ER.
PMCID: PMC3912727  PMID: 23703827
Medical Records Systems, Computerized [L01.700.508.300.695]; Medical Record Linkage [N04.452.859.564.550]
7.  A New Socio-technical Model for Studying Health Information Technology in Complex Adaptive Healthcare Systems 
Quality & safety in health care  2010;19(Suppl 3):i68-i74.
Conceptual models have been developed to address challenges inherent in studying health information technology (HIT). This manuscript introduces an 8-dimensional model specifically designed to address the socio-technical challenges involved in design, development, implementation, use, and evaluation of HIT within complex adaptive healthcare systems. The 8 dimensions are not independent, sequential, or hierarchical, but rather are interdependent and interrelated concepts similar to compositions of other complex adaptive systems. Hardware and software computing infrastructure refers to equipment and software used to power, support, and operate clinical applications and devices. Clinical content refers to textual or numeric data and images that constitute the “language” of clinical applications. The human computer interface includes all aspects of the computer that users can see, touch, or hear as they interact with it. People refers to everyone who interacts in some way with the system, from developer to end-user, including potential patient-users. Workflow and communication are the processes or steps involved in assuring that patient care tasks are carried out effectively. Two additional dimensions of the model are internal organizational features (e.g., policies, procedures, and culture) and external rules and regulations, both of which may facilitate or constrain many aspects of the preceding dimensions. The final dimension is measurement and monitoring, which refers to the process of measuring and evaluating both intended and unintended consequences of HIT implementation and use. We illustrate how our model has been successfully applied in real-world complex adaptive settings to understand and improve HIT applications at various stages of development and implementation.
PMCID: PMC3120130  PMID: 20959322
8.  Review of Reported Clinical Information System Adverse Events in US Food and Drug Administration Databases 
Applied clinical informatics  2011;2(1):63-74.
The US FDA has been collecting information on medical devices involved in significant adverse advents since 1984. These reports have been used by researchers to advise clinicians on potential risks and complications of using these devices.
Research adverse events related to the use of Clinical Information Systems (CIS) as reported in FDA databases.
Three large, national, adverse event medical device databases were examined for reports pertaining to CIS.
One hundred and twenty unique reports (from over 1.4 million reports) were found, representing 32 manufacturers. The manifestations of these adverse events included: missing or incorrect data, data displayed for the wrong patient, chaos during system downtime and system unavailable for use. Analysis of these reports illustrated events associated with system design, implementation, use, and support.
The identified causes can be used by manufacturers to improve their products and by clinical facilities and providers to adjust their workflow and implementation processes appropriately. The small number of reports found indicates a need to raise awareness regarding publicly available tools for documenting problems with CIS and for additional reporting and dialog between manufacturers, organizations, and users.
PMCID: PMC3175794  PMID: 21938265
Electronic Health Records; Information Systems; Mandatory Reporting; Medical Errors; United States Food and Drug Administration
11.  How context affects electronic health record-based test result follow-up: a mixed-methods evaluation 
BMJ Open  2014;4(11):e005985.
Electronic health record (EHR)-based alerts can facilitate transmission of test results to healthcare providers, helping ensure timely and appropriate follow-up. However, failure to follow-up on abnormal test results (missed test results) persists in EHR-enabled healthcare settings. We aimed to identify contextual factors associated with facility-level variation in missed test results within the Veterans Affairs (VA) health system.
Design, setting and participants
Based on a previous survey, we categorised VA facilities according to primary care providers’ (PCPs’) perceptions of low (n=20) versus high (n=20) risk of missed test results. We interviewed facility representatives to collect data on several contextual factors derived from a sociotechnical conceptual model of safe and effective EHR use. We compared these factors between facilities categorised as low and high perceived risk, adjusting for structural characteristics.
Facilities with low perceived risk were significantly more likely to use specific strategies to prevent alerts from being lost to follow-up (p=0.0114). Qualitative analysis identified three high-risk scenarios for missed test results: alerts on tests ordered by trainees, alerts ‘handed off’ to another covering clinician (surrogate clinician), and alerts on patients not assigned in the EHR to a PCP. Test result management policies and procedures to address these high-risk situations varied considerably across facilities.
Our study identified several scenarios that pose a higher risk for missed test results in EHR-based healthcare systems. In addition to implementing provider-level strategies to prevent missed test results, healthcare organisations should consider implementing monitoring systems to track missed test results.
PMCID: PMC4244393  PMID: 25387758
Elecronic Health Records; Missed Test Results; Patient Follow-up; Social-Technical Model; Patient Safety; Diagnostic Errors
12.  Assessing the Anticipated Consequences of Computer-based Provider Order Entry at Three Community Hospitals Using an Open-ended, Semi-structured Survey Instrument 
To determine what “average” clinicians in organizations that were about to implement Computer-based Provider Order Entry (CPOE) were expecting to occur, we conducted an open-ended, semi-structured survey at three community hospitals.
We created an open-ended, semi-structured, interview survey template that we customized for each organization. This interview-based survey was designed to be administered orally to clinicians and take approximately five minutes to complete, although clinicians were allowed to discuss as many advantages or disadvantages of the impending system roll-out as they wanted to.
Our survey findings did not reveal any overly negative, critical, problematic, or striking sets of concerns. However, from the standpoint of unintended consequences, we found that clinicians were anticipating only a few of the events, emotions, and process changes that are likely to result from CPOE.
The results of such an open-ended survey may prove useful in helping CPOE leaders to understand user perceptions and predictions about CPOE, because it can expose issues about which more communication, or discussion, is needed. Using the survey, implementation strategies and management techniques outlined in this paper, any chief information officer (CIO) or chief medical information officer (CMIO) should be able to adequately assess their organization's CPOE readiness, make the necessary mid-course corrections, and be prepared to deal with the currently identified unintended consequences of CPOE should they occur.
PMCID: PMC2668516  PMID: 17931963
Medical Order Entry Systems; Ethnology; Hospitals, Community; Medical Informatics
13.  Grand Challenges in Clinical Decision Support v10 
Journal of biomedical informatics  2007;41(2):387-392.
There is a pressing need for high-quality, effective means of designing, developing, presenting, implementing, evaluating, and maintaining all types of clinical decision support capabilities for clinicians, patients and consumers. Using an iterative, consensus-building process we identified a rank-ordered list of the top 10 grand challenges in clinical decision support. This list was created to educate and inspire researchers, developers, funders, and policy-makers. The list of challenges in order of importance that they be solved if patients and organizations are to begin realizing the fullest benefits possible of these systems consists of: Improve the human-computer interface; Disseminate best practices in CDS design, development, and implementation; Summarize patient-level information; Prioritize and filter recommendations to the user; Create an architecture for sharing executable CDS modules and services; Combine recommendations for patients with co-morbidities; Prioritize CDS content development and implementation; Create internet-accessible clinical decision support repositories; Use freetext information to drive clinical decision support; Mine large clinical databases to create new CDS. Identification of solutions to these challenges is critical if clinical decision support is to achieve its potential and improve the quality, safety and efficiency of healthcare.
PMCID: PMC2660274  PMID: 18029232
14.  Health Care Transformation Through Collaboration on Open-Source Informatics Projects: Integrating a Medical Applications Platform, Research Data Repository, and Patient Summarization 
The Strategic Health IT Advanced Research Projects (SHARP) program seeks to conquer well-understood challenges in medical informatics through breakthrough research. Two SHARP centers have found alignment in their methodological needs: (1) members of the National Center for Cognitive Informatics and Decision-making (NCCD) have developed knowledge bases to support problem-oriented summarizations of patient data, and (2) Substitutable Medical Apps, Reusable Technologies (SMART), which is a platform for reusable medical apps that can run on participating platforms connected to various electronic health records (EHR). Combining the work of these two centers will ensure wide dissemination of new methods for synthesized views of patient data. Informatics for Integrating Biology and the Bedside (i2b2) is an NIH-funded clinical research data repository platform in use at over 100 sites worldwide. By also working with a co-occurring initiative to SMART-enabling i2b2, we can confidently write one app that can be used extremely broadly.
Our goal was to facilitate development of intuitive, problem-oriented views of the patient record using NCCD knowledge bases that would run in any EHR. To do this, we developed a collaboration between the two SHARPs and an NIH center, i2b2.
First, we implemented collaborative tools to connect researchers at three institutions. Next, we developed a patient summarization app using the SMART platform and a previously validated NCCD problem-medication linkage knowledge base derived from the National Drug File-Reference Terminology (NDF-RT). Finally, to SMART-enable i2b2, we implemented two new Web service “cells” that expose the SMART application programming interface (API), and we made changes to the Web interface of i2b2 to host a “carousel” of SMART apps.
We deployed our SMART-based, NDF-RT-derived patient summarization app in this SMART-i2b2 container. It displays a problem-oriented view of medications and presents a line-graph display of laboratory results.
This summarization app can be run in any EHR environment that either supports SMART or runs SMART-enabled i2b2. This i2b2 “clinical bridge” demonstrates a pathway for reusable app development that does not require EHR vendors to immediately adopt the SMART API. Apps can be developed in SMART and run by clinicians in the i2b2 repository, reusing clinical data extracted from EHRs. This may encourage the adoption of SMART by supporting SMART app development until EHRs adopt the platform. It also allows a new variety of clinical SMART apps, fueled by the broad aggregation of data types available in research repositories. The app (including its knowledge base) and SMART-i2b2 are open-source and freely available for download.
PMCID: PMC3668611  PMID: 23722634
clinical information systems; medical informatics; knowledge bases; user-computer interface; data display; diffusion of innovation
15.  Beyond the threshold: real-time use of evidence in practice 
In two landmark reports on Quality and Information Technology, the Institute of Medicine described a 21st century healthcare delivery system that would improve the quality of care while reducing its costs. To achieve the improvements envisioned in these reports, it is necessary to increase the efficiency and effectiveness of the clinical decision support that is delivered to clinicians through electronic health records at the point of care. To make these dramatic improvements will require significant changes to the way in which clinical practice guidelines are developed, incorporated into existing electronic health records (EHR), and integrated into clinicians’ workflow at the point of care. In this paper, we: 1) discuss the challenges associated with translating evidence to practice; 2) consider what it will take to bridge the gap between the current limits to use of CPGs and expectations for their meaningful use at the point of care in practices with EHRs; 3) describe a framework that underlies CDS systems which, if incorporated in the development of CPGs, can be a means to bridge this gap, 4) review the general types and adoption of current CDS systems, and 5) describe how the adoption of EHRs and related technologies will directly influence the content and form of CPGs. Achieving these objectives should result in improvements in the quality and reductions in the cost of healthcare, both of which are necessary to ensure a 21st century delivery system that consistently provides safe and effective care to all patients.
PMCID: PMC3639800  PMID: 23587225
16.  Safety Assurance Factors for Electronic Health Record Resilience (SAFER): study protocol 
Implementation and use of electronic health records (EHRs) could lead to potential improvements in quality of care. However, the use of EHRs also introduces unique and often unexpected patient safety risks. Proactive assessment of risks and vulnerabilities can help address potential EHR-related safety hazards before harm occurs; however, current risk assessment methods are underdeveloped. The overall objective of this project is to develop and validate proactive assessment tools to ensure that EHR-enabled clinical work systems are safe and effective.
This work is conceptually grounded in an 8-dimension model of safe and effective health information technology use. Our first aim is to develop self-assessment guides that can be used by health care institutions to evaluate certain high-risk components of their EHR-enabled clinical work systems. We will solicit input from subject matter experts and relevant stakeholders to develop guides focused on 9 specific risk areas and will subsequently pilot test the guides with individuals representative of likely users. The second aim will be to examine the utility of the self-assessment guides by beta testing the guides at selected facilities and conducting on-site evaluations. Our multidisciplinary team will use a variety of methods to assess the content validity and perceived usefulness of the guides, including interviews, naturalistic observations, and document analysis. The anticipated output of this work will be a series of self-administered EHR safety assessment guides with clear, actionable, checklist-type items.
Proactive assessment of patient safety risks increases the resiliency of health care organizations to unanticipated hazards of EHR use. The resulting products and lessons learned from the development of the assessment guides are expected to be helpful to organizations that are beginning the EHR selection and implementation process as well as those that have already implemented EHRs. Findings from our project, currently underway, will inform future efforts to validate and implement tools that can be used by health care organizations to improve the safety of EHR-enabled clinical work systems.
PMCID: PMC3639028  PMID: 23587208
Electronic health records; Health information technology; Patient safety; Risk assessment; Resilience
17.  Use of a support vector machine for categorizing free-text notes: assessment of accuracy across two institutions 
Electronic health record (EHR) users must regularly review large amounts of data in order to make informed clinical decisions, and such review is time-consuming and often overwhelming. Technologies like automated summarization tools, EHR search engines and natural language processing have been shown to help clinicians manage this information.
To develop a support vector machine (SVM)-based system for identifying EHR progress notes pertaining to diabetes, and to validate it at two institutions.
Materials and methods
We retrieved 2000 EHR progress notes from patients with diabetes at the Brigham and Women's Hospital (1000 for training and 1000 for testing) and another 1000 notes from the University of Texas Physicians (for validation). We manually annotated all notes and trained a SVM using a bag of words approach. We then used the SVM on the testing and validation sets and evaluated its performance with the area under the curve (AUC) and F statistics.
The model accurately identified diabetes-related notes in both the Brigham and Women's Hospital testing set (AUC=0.956, F=0.934) and the external University of Texas Faculty Physicians validation set (AUC=0.947, F=0.935).
Overall, the model we developed was quite accurate. Furthermore, it generalized, without loss of accuracy, to another institution with a different EHR and a distinct patient and provider population.
It is possible to use a SVM-based classifier to identify EHR progress notes pertaining to diabetes, and the model generalizes well.
PMCID: PMC3756266  PMID: 23543111
electronic health record; support vector machine; natural language processing; search
18.  Emotional Aspects of Computer-based Provider Order Entry: A Qualitative Study 
Objectives: Computer-based provider order entry (CPOE) systems are implemented to increase both efficiency and accuracy in health care, but these systems often cause a myriad of emotions to arise. This qualitative research investigates the emotions surrounding CPOE implementation and use.
Methods: We performed a secondary analysis of several previously collected qualitative data sets from interviews and observations of over 50 individuals. Three researchers worked in parallel to identify themes that expressed emotional responses to CPOE. We then reviewed and classified these quotes using a validated hierarchical taxonomy of semantically homogeneous terms associated with specific emotions.
Results: The implementation and use of CPOE systems provoked examples of positive, negative, and neutral emotions. Negative emotional responses were the most prevalent, by far, in all the observations.
Conclusion: Designing and implementing CPOE systems is difficult. These systems and the implementation process itself often inspire intense emotions. If designers and implementers fail to recognize that various CPOE features and implementation strategies can increase clinicians' negative emotions, then the systems may fail to become a routine part of the clinical care delivery process. We might alleviate some of these problems by designing positive feedback mechanisms for both the systems and the organizations.
PMCID: PMC1205605  PMID: 15905478
19.  Improving the Effectiveness of Electronic Health Record-Based Referral Processes 
Electronic health records are increasingly being used to facilitate referral communication in the outpatient setting. However, despite support by technology, referral communication between primary care providers and specialists is often unsatisfactory and is unable to eliminate care delays. This may be in part due to lack of attention to how information and communication technology fits within the social environment of health care. Making electronic referral communication effective requires a multifaceted “socio-technical” approach. Using an 8-dimensional socio-technical model for health information technology as a framework, we describe ten recommendations that represent good clinical practices to design, develop, implement, improve, and monitor electronic referral communication in the outpatient setting. These recommendations were developed on the basis of our previous work, current literature, sound clinical practice, and a systems-based approach to understanding and implementing health information technology solutions. Recommendations are relevant to system designers, practicing clinicians, and other stakeholders considering use of electronic health records to support referral communication.
PMCID: PMC3492108  PMID: 22973874
20.  An analysis of electronic health record-related patient safety concerns 
A recent Institute of Medicine report called for attention to safety issues related to electronic health records (EHRs). We analyzed EHR-related safety concerns reported within a large, integrated healthcare system.
The Informatics Patient Safety Office of the Veterans Health Administration (VA) maintains a non-punitive, voluntary reporting system to collect and investigate EHR-related safety concerns (ie, adverse events, potential events, and near misses). We analyzed completed investigations using an eight-dimension sociotechnical conceptual model that accounted for both technical and non-technical dimensions of safety. Using the framework analysis approach to qualitative data, we identified emergent and recurring safety concerns common to multiple reports.
We extracted 100 consecutive, unique, closed investigations between August 2009 and May 2013 from 344 reported incidents. Seventy-four involved unsafe technology and 25 involved unsafe use of technology. A majority (70%) involved two or more model dimensions. Most often, non-technical dimensions such as workflow, policies, and personnel interacted in a complex fashion with technical dimensions such as software/hardware, content, and user interface to produce safety concerns. Most (94%) safety concerns related to either unmet data-display needs in the EHR (ie, displayed information available to the end user failed to reduce uncertainty or led to increased potential for patient harm), software upgrades or modifications, data transmission between components of the EHR, or ‘hidden dependencies’ within the EHR.
EHR-related safety concerns involving both unsafe technology and unsafe use of technology persist long after ‘go-live’ and despite the sophisticated EHR infrastructure represented in our data source. Currently, few healthcare institutions have reporting and analysis capabilities similar to the VA.
Because EHR-related safety concerns have complex sociotechnical origins, institutions with long-standing as well as recent EHR implementations should build a robust infrastructure to monitor and learn from them.
PMCID: PMC4215044  PMID: 24951796
Electronic Health Records; sociotechnical; reporting systems; medical errors; human factors; patients safety
21.  Clinical Decision Support Alert Appropriateness: A Review and Proposal for Improvement 
The Ochsner Journal  2014;14(2):195-202.
Many healthcare providers are adopting clinical decision support (CDS) systems to improve patient safety and meet meaningful use requirements. Computerized alerts that prompt clinicians about drug-allergy, drug-drug, and drug-disease warnings or provide dosing guidance are most commonly implemented. Alert overrides, which occur when clinicians do not follow the guidance presented by the alert, can hinder improved patient outcomes.
We present a review of CDS alerts and describe a proposal to develop novel methods for evaluating and improving CDS alerts that builds upon traditional informatics approaches. Our proposal incorporates previously described models for predicting alert overrides that utilize retrospective chart review to determine which alerts are clinically relevant and which overrides are justifiable.
Despite increasing implementations of CDS alerts, detailed evaluations rarely occur because of the extensive labor involved in manual chart reviews to determine alert and response appropriateness. Further, most studies have solely evaluated alert overrides that are appropriate or justifiable. Our proposal expands the use of web-based monitoring tools with an interactive dashboard for evaluating CDS alert and response appropriateness that incorporates the predictive models. The dashboard provides 2 views, an alert detail view and a patient detail view, to provide a full history of alerts and help put the patient's events in context.
The proposed research introduces several innovations to address the challenges and gaps in alert evaluations. This research can transform alert evaluation processes across healthcare settings, leading to improved CDS, reduced alert fatigue, and increased patient safety.
PMCID: PMC4052586  PMID: 24940129
Decision support systems–clinical; electronic health records; medical order entry systems; medication errors; prevention and control; reminder systems
22.  Recommended practices for computerized clinical decision support and knowledge management in community settings: a qualitative study 
The purpose of this study was to identify recommended practices for computerized clinical decision support (CDS) development and implementation and for knowledge management (KM) processes in ambulatory clinics and community hospitals using commercial or locally developed systems in the U.S.
Guided by the Multiple Perspectives Framework, the authors conducted ethnographic field studies at two community hospitals and five ambulatory clinic organizations across the U.S. Using a Rapid Assessment Process, a multidisciplinary research team: gathered preliminary assessment data; conducted on-site interviews, observations, and field surveys; analyzed data using both template and grounded methods; and developed universal themes. A panel of experts produced recommended practices.
The team identified ten themes related to CDS and KM. These include: 1) workflow; 2) knowledge management; 3) data as a foundation for CDS; 4) user computer interaction; 5) measurement and metrics; 6) governance; 7) translation for collaboration; 8) the meaning of CDS; 9) roles of special, essential people; and 10) communication, training, and support. Experts developed recommendations about each theme. The original Multiple Perspectives framework was modified to make explicit a new theoretical construct, that of Translational Interaction.
These ten themes represent areas that need attention if a clinic or community hospital plans to implement and successfully utilize CDS. In addition, they have implications for workforce education, research, and national-level policy development. The Translational Interaction construct could guide future applied informatics research endeavors.
PMCID: PMC3334687  PMID: 22333210
23.  Information Overload and Missed Test Results in EHR-based Settings 
JAMA internal medicine  2013;173(8):10.1001/2013.jamainternmed.61.
PMCID: PMC3822526  PMID: 23460235
diagnostic errors; patient safety; test results; care delays; health information technology; primary care; electronic health records; usability
24.  Towards successful coordination of electronic health record based-referrals: a qualitative analysis 
Successful subspecialty referrals require considerable coordination and interactive communication among the primary care provider (PCP), the subspecialist, and the patient, which may be challenging in the outpatient setting. Even when referrals are facilitated by electronic health records (EHRs) (i.e., e-referrals), lapses in patient follow-up might occur. Although compelling reasons exist why referral coordination should be improved, little is known about which elements of the complex referral coordination process should be targeted for improvement. Using Okhuysen & Bechky's coordination framework, this paper aims to understand the barriers, facilitators, and suggestions for improving communication and coordination of EHR-based referrals in an integrated healthcare system.
We conducted a qualitative study to understand coordination breakdowns related to e-referrals in an integrated healthcare system and examined work-system factors that affect the timely receipt of subspecialty care. We conducted interviews with seven subject matter experts and six focus groups with a total of 30 PCPs and subspecialists at two tertiary care Department of Veterans Affairs (VA) medical centers. Using techniques from grounded theory and content analysis, we identified organizational themes that affected the referral process.
Four themes emerged: lack of an institutional referral policy, lack of standardization in certain referral procedures, ambiguity in roles and responsibilities, and inadequate resources to adapt and respond to referral requests effectively. Marked differences in PCPs' and subspecialists' communication styles and individual mental models of the referral processes likely precluded the development of a shared mental model to facilitate coordination and successful referral completion. Notably, very few barriers related to the EHR were reported.
Despite facilitating information transfer between PCPs and subspecialists, e-referrals remain prone to coordination breakdowns. Clear referral policies, well-defined roles and responsibilities for key personnel, standardized procedures and communication protocols, and adequate human resources must be in place before implementing an EHR to facilitate referrals.
PMCID: PMC3199858  PMID: 21794109
25.  Understanding the management of electronic test result notifications in the outpatient setting 
Notifying clinicians about abnormal test results through electronic health record (EHR) -based "alert" notifications may not always lead to timely follow-up of patients. We sought to understand barriers, facilitators, and potential interventions for safe and effective management of abnormal test result delivery via electronic alerts.
We conducted a qualitative study consisting of six 6-8 member focus groups (N = 44) at two large, geographically dispersed Veterans Affairs facilities. Participants included full-time primary care providers, and personnel representing diagnostic services (radiology, laboratory) and information technology. We asked participants to discuss barriers, facilitators, and suggestions for improving timely management and follow-up of abnormal test result notifications and encouraged them to consider technological issues, as well as broader, human-factor-related aspects of EHR use such as organizational, personnel, and workflow.
Providers reported receiving a large number of alerts containing information unrelated to abnormal test results, many of which were believed to be unnecessary. Some providers also reported lacking proficiency in use of certain EHR features that would enable them to manage alerts more efficiently. Suggestions for improvement included improving display and tracking processes for critical alerts in the EHR, redesigning clinical workflow, and streamlining policies and procedures related to test result notification.
Providers perceive several challenges for fail-safe electronic communication and tracking of abnormal test results. A multi-dimensional approach that addresses technology as well as the many non-technological factors we elicited is essential to design interventions to reduce missed test results in EHRs.
PMCID: PMC3100236  PMID: 21486478
Decision Support Systems; Clinical; Automated notification; diagnostic errors; abnormal diagnostic test results; Medical Records Systems; Computerized; patient follow-up; patient safety; health information technology; communication; primary care

Results 1-25 (109)