Search tips
Search criteria 


Logo of nihpaAbout Author manuscriptsSubmit a manuscriptHHS Public Access; Author Manuscript; Accepted for publication in peer reviewed journal;
JAMA. Author manuscript; available in PMC 2010 October 28.
Published in final edited form as:
PMCID: PMC2965782

Safe Electronic Health Record use requires a comprehensive monitoring and evaluation framework

Recent passage of the American Reinvestment and Recovery Act (ARRA) increases pressure on healthcare practitioners and organizations to implement currently available electronic health records (EHRs). Research and experience gained to date shows that such implementation efforts are difficult, costly, time-consuming, and fraught with many unintended consequences1. Evaluation of these systems after implementation suggests that they do not routinely meet safety standards of other safety-critical industries2. The aggressive timeline proposed in the ARRA bill means that a large number of practitioners and organizations will soon be attempting a monumental feat without the time or ability to customize these systems to their local workflows3. In a previous article, we proposed 8 dimensions or “rights” of EHR safety that addressed the complex social, technical, and personal issues associated with EHR use4. The goal of this article is to describe an approach for a comprehensive EHR monitoring and evaluation framework (i.e., the 8th dimension of EHR safety). Without such a comprehensive framework, safe and effective EHR use cannot be assured. This proposed framework has five essential components: 1) ability for practitioners and organizations to report patient safety events or potential hazards related to EHR use 5,6; 2) enhanced EHR certification that includes specific assurances that good software development procedures7 have been followed along with evidence that previously reported adverse events and hazards have been addressed; 3) self-assessment, attestation, testing, and reporting by both clinicians and healthcare organizations that all eight dimensions of safe EHR use have been addressed; 4) Local, state, and national oversight in the form of an on-site, in-person accreditation of EHRs as implemented and used by clinicians in the healthcare setting; 5) A national EHR-related adverse event investigation board that reviews incident reports and has the authority to investigate.

Reporting EHR safety issues

Currently there is no clear organization or other entity for healthcare practitioners to report adverse events or potential safety hazards that result from inadequate design, development, implementation or use of EHRs. As Koppel and Kreda state, that many EHR vendors legally limit the ability of their clients to publically report these types of problems6. Such a national EHR hazard reporting system, as described in detail by Walker et al. could increase awareness of safety concerns among users as well as help vendors identify items that need their attention5. This reporting system might be operationalized through the new Patient Safety Organization Statute and associated Agency for Healthcare Research and Quality (AHRQ) common reporting formats8.

Enhanced EHR certification

The organizations responsible for certifying EHRs should add the following types of checks to their current certification criteria. All EHR vendors should have to demonstrate that they follow good software engineering practices including: performing hazard analyses of their products; designing for safety; documenting these designs; and verifying that their systems work as designed7. All systems should have their user interfaces tested for usability. All vendors should be prepared to demonstrate that they have successfully addressed all critical software issues identified within the national hazard reporting and investigation system within the previous year at any time. In addition, each EHR should be “load tested” to simulate its response time when multiple users are accessing the system simultaneously. Moreover, each vendor should be required to present data collected from multiple implementations describing their system’s reliability and response time as implemented. In conclusion, EHR vendors must demonstrate that their applications have been designed for safety, developed correctly, work as designed, and had all their defects fixed.

Self-Assessment of EHR use

A national, vendor-independent, federally-funded and managed EHR oversight organization should develop a self-assessment guide for all individual EHR users as well as the organizations responsible for implementing them. This guide for users should include at a minimum approximately 25 common actions that a user should be capable of performing (e.g., look up a patient by name or medical record number or review the 3 most recent laboratory test results); the organization's EHR downtime and reactivation procedures, and any EHR-related adverse events or potential hazards the user has been directly involved in. This self-assessment tool could be developed and implemented similarly to the way the AHRQ/LeapFrog EHR/clinical decision support assessment tool has been9.

In addition, each organization should perform and document a more extensive review of their clinical information systems on a yearly basis. This review should address each of the eight dimensions of safe EHR Use framework that includes: hardware and software; clinical content; user interfaces; user training and authorization procedures; clinical workflow and communication; organizational policies and procedures; compliance with state and federal rules and regulations; and periodic measurements of system activity.

The results of these self- and organizational assessments should be submitted to a central clearinghouse that would facilitate the creation and dissemination of national EHR safety benchmarks.

Onsite accreditation of EHRs as Implemented and Used

Although the first 3 proposals will most likely help to significantly improve the safety and effectiveness of EHR usage nationwide, there is still no substitute for periodic, unannounced, random, onsite inspections of EHR systems. These inspections could be conducted as part of ongoing organization-wide, quality and patient safety accreditation activities by organizations such as The Joint Commission or local state departments of health. Once again, these inspections should address all eight dimensions of Safe EHR use4. Organizations could be given six months to address all of the reviewers’ concerns or face significant financial penalties.

National EHR Adverse Event Investigation Board

As EHRs begin to play an increasingly important role in patient care on a large scale, community-wide basis, the Office of the National Coordinator for Health Information Technology (ONC) could create a National EHR Adverse Event Investigation Board similar to the National Transportation Safety Board or the Commercial Aviation Safety Team10, to investigate serious EHR-related adverse events or hazards that involve complex, sociotechnical interactions between clinicians or patients and software systems. The findings of these investigations could be reported to the ONC and made public so that patients, other vendors, healthcare organizations, and practitioners can learn from them. This board would be provided with unlimited access to all aspects of the EHR including for example: key system backups, change logs, and minutes from the local EHR safety oversight committees. This board would have the authority to investigate all serious EHR-related adverse events or near misses. In conclusion, President Obama has taken an important step toward improving the clinical computing infrastructure of the US healthcare delivery system by stating the goal of all citizens having access to an EHR. However, the extremely aggressive timeline in the ARRA stimulus package places enormous pressure on healthcare practitioners and their organizations to rapidly implement EHRs. Such rapid implementations could lead to significant patient safety events. To help ensure that these implementations are as safe and effective as possible, this 5-stage proposal to create a comprehensive EHR monitoring and evaluation framework should be considered. Using such a framework, the ONC, or another designated national body, could begin to provide the oversight needed to ensure that all EHR implementations are safe and effective and to provide the mechanisms to help healthcare organizations using these systems to deliver the highest quality, lowest cost, and safest healthcare possible.


Funding Source

Dr. Sittig is supported in part by grant R01- LM006942 from the National Library of Medicine. This source had no role in the preparation, review, or approval of the manuscript.


Conflicts of Interest

Dean F. Sittig: None

David C. Classen is an employee and stockholder of CSC a technology services company.


1. Sittig DF, Ash JS. Clinical information systems: Overcoming adverse consequences. Sudbury, MA: Jones and Bartlett; 2009.
2. Bhansali PV. Universal software safety standard. SIGSOFT Softw Eng Notes. Sep, 2005. pp. 1–4. Available at:
3. Sittig DF, Ash JS, Jiang Z, Osheroff JA, Shabot MM. Lessons from “unexpected increased mortality after implementation of a commercially sold computerized physician order entry system” Pediatrics. 2006 Aug;118(2):797–801. [PubMed]
4. Sittig DF, Singh H. Eight Rights of Safe Electronic Health Record Use. Journal of American Medical Association. 2009 Sep 9;302(10):1111–3. [PubMed]
5. Walker JM, Carayon P, Leveson N, Paulus RA, Tooker J, Chin H, Bothe A, Jr, Stewart WF. EHR safety: the way forward to safe and effective systems. J Am Med Inform Assoc. 2008 May–Jun;15(3):272–7. [PMC free article] [PubMed]
6. Koppel R, Kreda D. Health care information technology vendors' “hold harmless” clause: implications for patients and clinicians. JAMA. 2009 Mar 25;301(12):1276–8. [PubMed]
7. Leveson N. Safeware: System Safety and Computers. Indianapolis, IN: Addison Wesley; 1995.
8. Sanghvi AA. Patient safety and quality improvement act finalized. J Med Assoc Ga. 2009;98(1):53–4. [PubMed]
9. Kilbridge PM, Welebob EM, Classen DC. Development of the Leapfrog methodology for evaluating hospital implemented inpatient computerized physician order entry systems. Qual Saf Health Care. 2006 Apr;15(2):81–4. [PMC free article] [PubMed]
10. Pronovost PJ, Goeschel CA, Olsen KL, Pham JC, Miller MR, Berenholtz SM, Sexton JB, Marsteller JA, Morlock LL, Wu AW, Loeb JM, Clancy CM. Reducing health care hazards: lessons from the commercial aviation safety team. Health Aff (Millwood) 2009 May–Jun;28(3):w479–89. [PubMed]