PMCCPMCCPMCC

Search tips
Search criteria 

Advanced

 
Logo of nihpaAbout Author manuscriptsSubmit a manuscriptHHS Public Access; Author Manuscript; Accepted for publication in peer reviewed journal;
 
Int J Med Inform. Author manuscript; available in PMC 2011 January 1.
Published in final edited form as:
PMCID: PMC2895508
NIHMSID: NIHMS148030

The state of the art in clinical knowledge management: An inventory of tools and techniques

Dean F. Sittig, Ph.D.,1 Adam Wright, Ph.D.,2 Linas Simonaitis, M.D.,3 James D. Carpenter, R.Ph., M.S.,4 George O. Allen,5 Bradley N. Doebbeling, MD, MSc,3,5 Anwar Mohammad Sirajuddin, M.B.B.S, M.S.,6 Joan S. Ash, Ph.D., M.B.A., M.L.S.,7 and Blackford Middleton, MD, MPH, MSc2

Abstract

Purpose

To explore the need for, and use of, high-quality, collaborative, clinical knowledge management (CKM) tools and techniques to manage clinical decision support content.

Methods

In order to better understand the current state of the art in CKM, we developed a survey of potential CKM tools and techniques. We conducted an exploratory study by querying a convenience sample of respondents about their use of specific practices in CKM.

Results

The following tools and techniques should be priorities in organizations interested in developing successful computer-based provider order entry (CPOE) and clinical decision support (CDS) implementations: 1) A multidisciplinary team responsible for creating and maintaining the clinical content; 2) An external organizational repository of clinical content with web-based viewer that allows anyone in the organization to review it; 3) An online, collaborative, interactive, internet-based tool to facilitate content development; 4) An enterprise-wide tool to maintain the controlled clinical terminology concepts. Even organizations that have been successfully using Computer-based Provider Order Entry with advanced Clinical Decision Support features for well over 15 years are not using all of the CKM tools or practices that we identified.

Conclusions

If we are to further stimulate progress in the area of clinical decision support, we must continue to develop and refine our understanding and use of advanced CKM capabilities.

Introduction

Since the Institute of Medicine’s release of “To Err is Human: Building a Safer Health System” in 1999 [1] and “Crossing the Quality Chasm: A New Health System for the 21st Century” in 2001 [2], there has been increasing pressure on everyone from healthcare providers working in solo practices to large nationwide, healthcare delivery networks to implement state of the art clinical information systems. Recent reports have reiterated the fact that, for these clinical information systems to deliver their expected improvements in quality, they must be equipped with state of the art, real-time, point-of-care, clinical decision support (CDS) features [3]. Unfortunately, the successful design, development, implementation, maintenance and evaluation of these advanced clinical decision support features is in itself a complex undertaking that is not well understood [4]. These advanced clinical decision support systems generally depend on large knowledge bases of clinical knowledge. For example, a drug-drug interaction checker requires a database of drug names and their interactions. This database must be kept up-to-date as new drugs become available, as new drug interactions become known and as our understanding of existing interactions evolves. Manual systems can be used for this task when the amount of clinical knowledge to manage is small (i.e., less than 100 interactions), but as the size of these databases grows, clinical knowledge management (CKM) tools become necessary to ensure that these clinical knowledge bases are correct, consistent, complete and current. In fact, the area of research and development referred to as “clinical knowledge management” will soon become vitally important as organizations struggle to develop, implement, maintain, and evaluate their clinical decision support efforts.

The need for high quality, collaborative, CKM practices is often overlooked even in the most advanced healthcare organizations [5]. The need to develop high quality, evidence-based CDS interventions, coupled with the need to obtain widespread, peer review of this work, makes the use of an internet-based, collaborative, CKM tool almost mandatory. Moreover, this clinical content must be kept up to date as the existing knowledge changes, new clinical knowledge is gained, and underlying changes take place in the clinical information system [6]. This places enormous burdens on the informaticians responsible for maintaining all of the clinical applications that rely on this knowledge including: Computer-based Provider Order Entry (CPOE), clinical results review, clinical documentation, and medication administration systems, for example.

In a recent article, we identified “disseminate best practices in Clinical Decision Support (CDS) design, development, and implementation” as one of the “grand challenges” in CDS [7]. A key component of any CDS initiative is how the organization develops, disseminates, maintains, and evaluates its clinical knowledge content, or its Clinical Knowledge Management practices [8].

In an attempt to explore the current state of the art in CKM, we developed an extensive survey of potential CKM tools and techniques (see Appendix). This manuscript describes the content of the CKM survey and reports our initial findings from administering it to a convenience sample of six organizations with substantial experience in the development and use of advanced clinical information systems with state-of-the-art, real-time, clinical decision support systems.

Methods

Survey Development and Testing

We developed a survey of clinical knowledge management tools and techniques, which we call the clinical knowledge management inventory, based on our extensive review of the CPOE and CDS-related literature (see [9,10] for links to an online bibliography of these two domains), interviews with experts and our own experiences in working in the field for several decades. Please see Table 2 for a list of questions.

Table 2
Questions used to identify current practices in clinical knowledge management.

Survey Site Selection

We identified six geographically (2 Northwest, 2 Midwest, 1 Southwest, and 1 Northeast) and organizationally (a mid-size community hospital, a large nationwide health maintenance organization, two large integrated delivery networks, a county hospital affiliated with an academic medical center, and a Veteran’s Administration hospital) diverse healthcare organizations. Four of the six have been recognized for their excellence in implementation and use of advanced clinical information systems including use of clinical decision support features and functions [11,12]. In addition we sought to identify sites representing healthcare organizations with both commercially acquired and internally developed clinical information systems.

Survey Administration

Following Institutional Review Board (IRB) approval, we pre-tested the CKM site inventory with two individuals in the Portland, Oregon area. After refining several questions, we then distributed it, via email, to a single individual at each of the study sites who was directly involved in CPOE / CDS and CKM activities. If this individual had any specific clarifying questions regarding particular survey items, we answered them promptly via email or telephone.

Results

We have divided the results into three sections. In the first section, we present characterizations of the six organizations’ use of computer-based provider order entry (CPOE), various CDS intervention types, and their clinical knowledge management activities. The second section describes the content of the CKM site inventory including: 1) an overview of the CKM tools we identified as keys to success and 2) questions we asked at each site to help us better understand current practices in clinical knowledge. In the third section, we present our findings from administering this survey within six disparate organizations.

Results: 1. Characterizations of the six organizations

Three of the six sites used commercially available clinical information system products (McKesson, San Francisco, CA; Epic Systems, Verona, WI; Cerner, Kansas City, MO) while three used in-house designed and developed systems. The sites had been live with their CPOE / CDS systems from 2 – 22 years (median: 13 years). The number of order sets each site had in their production environment ranged from 185 – 801 (median: approximately 325) while the number of alerts in the production environment ranged from 40 – 7,120 (median: less than 100). In terms of the number of people (e.g., clinical domain experts, knowledge engineers, nurses, evidence-based medicine experts, informaticians, information technology professionals) involved in discussions regarding specific clinical decision support interventions, the range was between 5 and 15 people (median: 7).

Results: 2. Content of the Clinical Knowledge Management (CKM) Site Inventory

Through interviews with experts and a review of literature, we identified four tools that appear to be useful for CKM. We describe these tools in the following sections.

Tool 1: External repository of clinical content with web-based viewer

Clinical decision support content is often embedded directly in the logic of clinical information systems. This can be expedient, but, when intermingled, the content can be much more difficult to locate and maintain. While only a few of the organizations we surveyed have fully implemented such an external repository, all agreed that the ideal practice is to maintain the clinical knowledge separate from the code that implements the information system in an external repository that is both browsable and searchable. Often this repository consists of either a database or file storage system with a dedicated application interface or a browser-based interface. The key here is not the format or access mechanism for the knowledge, but, rather that storing the knowledge in an external repository, rather than just the clinical information system, facilitates the ability of clinicians to maintain, share, and review the content. With such a system, a knowledge engineer can quickly answer the question, “What is our organization doing to address the Joint Commission’s new Venous Thromboembolism (VTE) Core Measure [13]?” by simply searching the repository for all CDS content containing the term “venous thromboembolism”.

Tool 2: Online, collaborative, interactive, intranet-based tool to facilitate content development

Developing or modifying new clinical decision support content within an organization is always a challenge. A variety of people are usually involved in content development, including physicians, nurses, pharmacists, informaticians, software developers and quality improvement professionals. Bringing all of these participants together for in-person meetings is challenging. Allowing content developers and users to have an asynchronous discussion regarding the pros and cons of specific CDS interventions can be a valuable method of both developing new clinical content and gaining organizational consensus. Many organizations are beginning to experiment with new Web 2.0 technologies such as wikis, discussion boards, or listservs to facilitate distributed and asynchronous creation, consensus-building, and review of clinical content [14]. These tools are enabling such organizations to include more people in the conversation and to reach consensus more quickly than ever before.

Tool 3: Enterprise-wide tool to maintain controlled clinical terminology concepts

One of the most important prerequisites to the development and use of advanced, patient-specific CDS features and functions is the availability of a robust, controlled clinical terminology (e.g., SNOMED for problems, LOINC for lab results andICD-9 for billing diagnoses, etc.). Without such a controlled clinical terminology, CDS developers will have difficulty using the “freetext” clinical concepts within their logic. In addition, many controlled terminologies include some sort of semantic network that maintains various types of relationships among the clinical terms. These relationships can greatly simplify the construction and maintenance of the CDS. For example, a knowledge engineer might want to encode the following logic, “if the patient is on an ACE inhibitor, then…”. Without a controlled terminology that includes hierarchical relationships, the knowledge engineer would be required to list all of the Angiotensin Converting Enzyme inhibitors (i.e., ACE inhibitors) currently in use within the organization, and then update this list whenever a change was made to the organization’s formulary. Using such a tool (see figure 4), a knowledge engineer would simply add the new medication name, or a synonym for it, to the terminology and note that it is a “child” of the term ACE inhibitors. Then in the rule logic, the knowledge engineer would call the “ACE inhibitor query” (i.e., rather than listing all the potential ACE inhibitors by name) which would find all the medications in the ACE inhibitor category and compare the patient’s current medications to that list and return TRUE if one exists, FALSE otherwise.

Figure 4
Screen print showing the tool that Kaiser Permanente uses to manage term relationships within their convergent medical terminology. (Courtesy of Robert H. Dolin, MD)

Tool 4: Tool for CDS users to provide feedback regarding specific CDS interventions

No matter how hard an organization works prior to deployment of new CDS interventions, there will always be clinicians who find justifiable exceptions to any type of CDS (e.g., alert, order set, condition-specific results display, default medication dose, etc.) developed. In addition to regularly scheduled meetings and clinical analysts who frequently interact with the clinicians at their work site, many organizations have developed a simple on-line method of gathering feedback from users while they are actually using the CDS intervention. These systems often make use of commonly available computer-based communication tools such as email, discussion groups, or databases. Often these tools capture some basic information about the user, the patient, and the clinical context in which they were working in addition to the user’s complaint(s) or suggestions. It is critical that organizations have an organized and efficient means of responding to these “suggestions” or “complaints”.

Current practices for clinical knowledge management

In addition to asking about the availability and use of CKM tools, our site inventory also asked participants about their use of specific practices for CKM. We developed this list (see Table 2) based on an extensive review of the literature surrounding clinical decision support development and maintenance in addition to interviews with experts in the field and observations conducted at many of the leading academic medical centers in the United States. While we do not yet have randomized clinical trial level evidence that any of these features, functions, or organizational practices results in higher quality clinical decision support, our experience leads us to believe that these ten items are at the very least key issues to consider when developing and maintaining high-quality clinical decision support systems.

Results: 3. Findings from administering the CKM site inventory

The first part of the CKM site inventory asked organizations to describe whether they had any of the following CKM tools (See Table 3). If they responded positively, we than asked them to describe how they were using each tool.

Table 3
A List of clinical knowledge management tools available at each participating institution.

Tool 1: External repository of clinical content with web-based viewer

These clinical knowledge repositories are being used in two different ways at four of the six sites. For example, at Partners, the CKM repository (see figure 1) documents the results of the on-line and in-person discussions and agreements regarding the proposed clinical content. The content of the CKM repository is then used by the programmer/analysts as they manually implement the logic within their clinical information system. On the other hand, within Kaiser Permanente Northwest, content is created directly in the clinical information system and the CKM repository is built by extracting the clinical knowledge from the clinical information system into a intranet-accessible directory which can be viewed, searched, or browsed via a simple web interface (see figure 2). Both of these methods of creating a CKM can create synchronization problems. For example at Partners, the clinical programmer/analysts may have to make slight modifications to the logic when they implement it within the CIS. If they do not simultaneously make these changes to the documents contained in the CKM, a synchronization problem occurs. Likewise, with the Kaiser Permanente Northwest solution, the content of the CKM is only as current as the last extraction. As shown in figure 2, which was created on August 26, 2008, the content was last extracted on August 14, 2008. While 12 days may not seem like a long time lag, it is quite possible that the clinical content may have changed since the last update. Again, the point is that any manually maintained external repository of clinical content can easily become out of sync with what is being used in the production-level clinical information system.

Figure 1
Screen print showing the search interface to the Partners Knowledge Management Portal.
Figure 2
Screen print showing table of contents of all clinical knowledge that has been extracted from the clinical information system.

Tool 2: Online, collaborative, interactive, intranet-based tool to facilitate content development

All, but one, of the sites are using some sort of on-line, collaborative, intranet-based tool to facilitate content development. Figure 3 shows a recent example use of Partners’ eRooms for discussion of how the outpatient EHR should behave when a clinician indicates that a patient has had a colonoscopy examination when the patient reports that he/she had it “done elsewhere”. The developers of their electronic medical record system had received complaints from end users who felt they were being inundated with “false-positive” reminders. The Partners Knowledge Management staff created a new discussion area within eRoom and invited members of the primary care subject matter expert panel to share their thoughts. Users discussed the various reminder intervals and participated in a poll. Ultimately, the decision was made to delay the reminder for 5 years in the event that either a colonoscopy or sigmoidoscopy had been completed and for 1 year if the patient had a stool guaiac test performed.

Figure 3
Example of an online, guided discussion in Partners’ eRoom (Note: Clinician identifying information has been obscured for privacy.)

Tool 3: Enterprise-wide tool to maintain controlled clinical terminology concepts

As mentioned above, controlled clinical terminology is critical to successful implementation and maintenance of large-scale (i.e., more than 100 rule-based alerts or over 100 order sets) clinical decision support operations [15, 16]. In order to maintain such a terminology, four of the six organizations have implemented sophisticated computer-based tools that allow knowledge engineers and terminologists to create new terms and manage the relationships between these terms as well as hierarchical concept groupings (see [17] for a more complete list of requirements for these tools). Figure 4 shows a screen print of Kaiser Permanente’s tool for managing their convergent medical terminology [18]. In addition to this “in-house” developed example, there are several commercially available products that are available, however none of the organizations we studied were currently using any of these products, which include: Apelon (Ridgefield, CT) with their Distributed Terminology System (DTS) which recently became available as an open-source product [19]; Intelligent Medical Objects (Northbrook, IL) IMO® Health Objects: Clinical Interface Terminology; and Health Language® (Aurora, CO) (HLI) Language Engine™. The reason that none of the organizations we surveyed were using any of these commercially available terminology maintenance tools is based on their history and not from any failing on the part of these vendors. The four organizations with locally developed tools are among the early leaders in the field of clinical decision support and they identified the need to have these sorts of tools long before any commercially available tools were available. To date, they have all made the decision that the benefits of moving to one of these new commercially available tools are outweighed by the costs in money, time, effort, and potential loss of idiosyncratic functions that switching would inevitably entail. Finally, many of the commercial electronic medical record vendors either directly provide or collaborate with companies like those listed above to offer very similar types of tools.

Tool 4: Tool for CDS users to provide feedback regarding specific CDS interventions

All of the sites surveyed had on-line tools to facilitate the collection and analysis of user feedback. Figure 5 shows an alert for Deep Venous Thrombosis prophylaxis from the Brigham & Women’s Hospital in Boston, MA. If the user wishes to communicate with the system developers regarding this alert, they can click on a <Comment> button which brings up a feedback screen. In this instance the developers have created both a structured as well as a freetext feedback form. All feedback is sent directly to the specific “rule owner” for each rule.

Figure 5Figure 5
Top: Shows an alert for Deep Venous Thrombosis prophylaxis. Clicking on the >Comment< button at bottom right of the Alert screen (top) brings up theAlert Evaluation screen (bottom). Input to section 1 (check boxes) and section 2 (small ...

Current practices for clinical knowledge management

The final portion of the CKM site inventory asked participants about their use of specific practices for CKM.

Question 1

Do you have a dedicated, multidisciplinary team of individuals responsible for creating and maintaining your clinical knowledge and clinical decision support interventions?

All organizations had such a team and all agreed that these individuals were the most essential component of their CDS success. In addition, they stated that a transparent governance structure for all content-related decision making was also important. They felt that transparent governance structures were necessary to ensure that all relevant stakeholders had a say in the final version of the content. Further, transparency was important when consensus could not be achieved. This became more common as the scale of the organization grew, for example in disagreements between individual provider vs. practice/clinic level vs. institution vs. integrated delivery network; and in the future, regional and national networks.

Question 2

Do you modify the external clinical knowledge (content) you use before implementing it? If so, how?

Many organizations purchase or otherwise bring in externally a portion of the clinical knowledge they use. In our surveys, it appeared that drug knowledge bases were the most widely purchased assets. However, the current state of the art in purchasing or sharing external clinical knowledge requires at least a minimal, and potentially, a great deal of modification. In addition, once the content is deemed acceptable by the local organization, it must still be translated into a computable form (e.g., the Gopher Care language in use at Regenstrief or Epic’s internal clinical decision support structures) by local physicians, nurses, or pharmacists with informatics training or sometimes informaticians without specific clinical training or experience. Some sites reported making substantial modifications to the clinical content, for example, selecting a small subset of the most clinically significant drug-drug interactions for use in alerting [20].

Question 3

Do you store your clinical knowledge externally using one of the existing clinical knowledge representation standards (Arden, GLIF, GEM [21])?

None of the sites contacted in this survey used any of these standard representation formats.

Question 4

Does your clinical information system have the capability of using one of the knowledge representation standards (Arden, GLIF, InfoButton, etc.) to execute its clinical decision support?

None of the sites contacted in this survey used any of these representation standards. There were many reasons for this including: the CIS vendors did not support these standards; the standards were lacking in one or more key features that the organization felt was critical for their success [22], and many sites had a significant number of similarly functioning clinical decision support interventions developed within their systems before any of these relatively new standards were developed (see for example, McDonald, 1976 [23]. Furthermore, several organizations have implemented “infobutton-like” functionality by hard-coding uniform resource locators (URLs) into their system to provide users with key patient-specific or context-sensitive links to reference information directly from the screen that presents the patient’s current medications (though none are using the HL7 Infobutton manager standard) [48].

Question 5

What types of people do you have to help manage your clinical knowledge?

All of the organizations studied had one or more of the following types of people involved in the CKM process:

  • Pharmacists with formal informatics training (e.g., Masters or Doctorate in Medical Informatics or Informatics fellowship) or extensive clinical informatics experience to develop and maintain pharmacy content
  • Physicians with informatics experience to translate clinical guidelines and study protocols into CDS interventions;
  • Doctoral-level medical informaticians.
  • Registered nurses (RNs) with informatics experience;
  • Dedicated software developers and project managers without a clinical background.

Question 6

Does each piece of clinical content (i.e., alert, order set, and patient education material) have an individual responsible for monitoring the underlying clinical knowledge and maintaining the CDS intervention?

All organizations used some sort of distributed CKM maintenance system. For example, all organizations had a pharmacist informatician responsible for developing and maintaining the content related to medications (indications, interactions, side effects, monitoring, default dosing, preferred medications and formulations). In addition, they each had physician informaticians with broader clinical oversight roles.

Often content specific to a particular clinical department (e.g., procedure or problem-based order sets) is managed by a clinical champion or medical informatician in that department.

Question 7

Can your system send automatic reminders that content review date is approaching to individuals responsible for maintaining the content?

Even though none of the organizations surveyed currently had this capability, all of them recognized the problems associated with keeping their clinical content current.

Question 8

Do you have a system in place to send automatic notifications to key stakeholders when clinical guidelines underlying CDS change?

Only one organization, which contracted with an external organization, Zynx, had this capability, though use of this functionality has been very limited. On the other hand, as the reliance on clinical decision support imbedded within clinical information systems increases all organizations realize that monitoring the clinical guidelines upon which the decision support is based will become even more important. Our experience has been that finding the resources to continuously monitor and maintain CDS interventions can tax even the most careful and financially secure organizations. Over the next several years we expect that more and more organizations will begin “out-sourcing” this work to commercial organizations that designed specifically to address this issue.

Question 9

Do you have a system in place to “learn” new clinical knowledge from your patient database?

None of the organizations in this study had this capability in place, although several have conducted specific research projects to begin experimenting with this type of capability [24]. Once again, over the next several years, most of the organizations expect this capability to become more common as they continue to develop large clinical data repositories and their informatics’ researchers develop the techniques necessary to accomplish this task [25].

Question 10

Do you purchase, or share, clinical knowledge from external sources, e.g., reference information, drug information (dose ranges, interactions, patient handouts, indications, etc.), order sets, alerts / reminders?

All sites except for one relied on an external drug information database for medication-related information. All sites surveyed had sophisticated internal systems in place to develop and manage CDS knowledge in response to clinical user feedback, as well as guidelines published in the medical and pharmacy literature, although we do not think that these practices are common at other organizations. All organizations either were using, or were considering the purchase of externally developed clinical order sets.

All of the sites also reported involvement in one or more efforts to share their content externally with other clinical sites or the general public. These sharing efforts included federally funded contracts to develop systems to share data within a small group of consortium members [for example, 26] as well as efforts to share content with other organizations using the same clinical information system vendor [14].

Discussion

We identified a variety of tools and current practices for CKM. After reviewing all of the responses to our survey and discussing the summarization of the data, we have identified the following tools and practices as the most widely used in organizations with successful CPOE and CDS implementations:

  1. A multidisciplinary team responsible for creating and maintaining the clinical content.
  2. An external repository of clinical content with web-based viewer that allows anyone to review it.
  3. An online, interactive, internet-based tool to facilitate content development and collaboration.
  4. An enterprise-wide tool to maintain the controlled clinical terminology concepts. Although these tools and practices are the most widely used, support was also seen for many of the other practices and tools we identified, even if adoption of them was, at present, more limited. We anticipate that use of these tools will increase over time.

Challenges and Opportunities for Improving Clinical Knowledge Management Capabilities

In addition to the issues surrounding the general lack of use of existing tools to support the work of clinical knowledge management, there are also several organizational issues that must be addressed. For example, the lack of money to hire additional appropriately trained clinical informaticians to serve as knowledge engineers results in existing personnel being forced to work in unfamiliar territory without the necessary time or understanding to “do the job right”. Also, involvement of users who are clinical specialists on content development teams is often hard to maintain without sufficient monetary incentives. Further, the rapidly expanding regulatory reporting and compliance requirements along with increasing emphasis on quality measures (e.g., The Joint Commissions’ CORE Measures [27] and National Patient Safety Goals [28] or Medicare’s Physician Quality Reporting Initiative (PQRI) [29], to name just a few) are placing tremendous demands on already over-extended CDS design and development human resources.

Finally, many organizations are having difficulty encouraging those working to develop clinical decision support interventions to begin using their new “web 2.0” advanced collaboration technologies. These workers continue to rely on email, with all its known problems, as their primary method for collaboration. This continued reliance on email hinders the ability of multiple workers to efficiently and effectively collaborate since some people continue to carry out discussions in emails, while other people carry out discussions on wikis, which leads to parallel systems of communication – which is more confusing, not less!

Taken together these socio-technical challenges and opportunities for improving the use of clinical knowledge management tools and techniques that we have identified lend further support to the work of many socio-technical-oriented knowledge management researchers in the business community [30]. For example, Damodaran et al. found that “the development and systematic application of effective techniques for addressing the ‘soft issues’ (i.e. the human, cultural and organizational issues) relevant to knowledge management systems” are critical to their widespread use and ultimate success [31]). Furthermore Hlupic et al. identified an additional area that must be addressed for successful implementation and use of knowledge management tools and techniques, namely philosophical or ‘abstract’ issues (e.g., fundamental ontological and epistemological issues about the notion and approach to the study of knowledge management) in addition to the technical ‘hard’, and organizational ‘soft’ issues [32].

Future Work

Clearly this study represents only a small sample of all organizations involved in clinical decision support and hence clinical knowledge management activities. The next steps are to conduct a similar survey across a wider range of institutions. Then, if these currently identified practices are shown to be widespread and useful, it would be interesting to attempt to carry out a series of clinical trials to assess the impact of the use of these tools and techniques on the overall success of an organization’s clinical decision support program.

Conclusion

The field of CKM is in its infancy, yet holds great promise. Even within organizations that have been successfully using CPOE with advanced CDS features for well over 15 years, we did not find any site that was using all of the practices in CKM that we identified. If we are to speed progress in the area of clinical decision support, we must continue to develop and refine our understanding, implementation and use of advanced clinical knowledge management capabilities.

Summary Table

What was already known on the topic?

  • For clinical information systems to deliver their expected improvements in health care quality they must be equipped with state of the art, real-time, point-of-care, clinical decision support (CDS) features.
  • The successful design, development, implementation, maintenance and evaluation of these advanced clinical decision support features is a complex undertaking that is not well understood.
  • The need for high quality, collaborative, clinical knowledge management tools and practices is often overlooked even in the most advanced healthcare organizations.

What this study added to our knowledge?

  • The following tools and techniques should be priorities in organizations interested in developing successful computer-based provider order entry (CPOE) and clinical decision support (CDS) implementations:
    • A multidisciplinary team responsible for creating and maintaining the clinical content.
    • An external repository of the organization’s clinical content with a web-based viewer that allows anyone in the organization to review it.
    • An online, collaborative, interactive, internet-based tool to facilitate content development.
    • An enterprise-wide tool to maintain the controlled clinical terminology concepts.
Table 1
Healthcare organizations involved in the clinical knowledge assessment study.

Acknowledgements

This work was funded in part by NLM Research Grant R56-LM006942 and AHRQ Contract HHHSA29020080010. The sponsors were not involved in the design, conduct or analysis of the study, or in the preparation of this manuscript.

Role of the funding source The funding agencies had no part in this manuscript.

The state of the art in clinical knowledge management: An inventory of tools and techniques

Types of clinical decision support to consider:

  • Order sets
  • Alert logic and suggestions
  • Clinical reference information
  • Patient education information

Description of Knowledge management features:

External repository of clinical content

The clinical knowledge used to create the clinical decision support within the clinical information system should be maintained in an external repository. This repository could consist of either a database with an application interface or as a collection of files stored in a dedicated hierarchical directory structure and accessible through a browser-based interface. By storing the knowledge in an external repository, rather than only in the clinical information system, the organization can more easily maintain, share, and allow clinicians to review the content.

Online, collaborative, interactive, internet-based tool to facilitate content development

Developing new clinical decision support content within an organization is always a challenge. Allowing content developers and users to have an asynchronous discussion regarding the pros and cons of specific CDS interventions can be a valuable method of both developing new clinical content and gaining organizational consensus. Many organizations are beginning to experiment with new Web 2.0 technologies such as wiki, discussion boards, or listservs to facilitate distributed creation, consensus-building, and review of clinical content

Enterprise-wide tool to maintain controlled clinical terminology concepts

One of the most important prerequisites to the development and use of advanced, patient-specific CDS features and functions is availability of a robust, controlled clinical terminology (e.g., SNOMED, LOINC, ICD-9, etc.). In order to maintain such a terminology, many organizations have implemented sophisticated computer-based tools that allow knowledge engineers to create new terms and manage the relationships and hierarchical groupings of these concepts.

Tool for CDS users to provide feedback regarding specific CDS interventions

No matter how hard an organization works prior to deployment of new CDS interventions, there will always be clinicians who find justifiable exceptions to any rule developed. Many organizations have developed a simple on-line method of gathering feedback from users while they are actually using the CDS intervention. These systems often make use of commonly available computer-based communication tools such as email, discussion groups, or databases. Often these tools capture some basic information about both the user, the patient, and the clinical context in which they were working in addition to the user’s complaint(s).

Web-based clinical content viewer

Even organizations that do not have an interactive tool to facilitate CDS content development often have a web-based tool that allows interested clinicians and administrators to review all clinical decision support interventions outside of the clinical information system. Many times these tools simply provide a browser front end to a database of clinical content that was automatically extracted from the clinical information system.

Knowledge Management Tools Available…What does your organization have?
External repository of clinical content
Online, collaborative, interactive, internet-based tool (e.g., Web 2.0
technologies such as wiki, discussion boards, or listservs) to facilitate
distributed creation, consensus-building, and review of clinical content
Enterprise-wide tool to maintain controlled clinical terminology concepts,
relationships, and hierarchical groupings
Tool or method to allow CDS users to provide feedback regarding specific
CDS interventions (e.g., email, discussion group, database, etc.)
Web-based clinical content viewer
Measures of Clinical Content & Knowledge Management System UseWhat does your organization have?
Percentage of clinical content reviewed in last year
 Order sets
 Alert logic and suggestions
 Reference information
 Patient Education information
Regular reports of clinical content usage available
Number of order sets available for use
Number of alerts in production
Ad hoc query facility available
Number of users who login to the KM system each week (or month or
quarter)
Median number of participants in specific CDS intervention discussions
Best Practices for Clinical Knowledge ManagementWhat does your organization do?
Do you modify the external clinical knowledge (content) you use before
implementing it? If so, how?
Do you store your clinical knowledge externally using one of the
knowledge representation standards (Arden, GLIF, GEM, MEL, etc.)?
Does your clinical information system have the capability of using one of
the knowledge representation standards (Arden, GLIF, InfoButton, etc.) to
execute its clinical decision support?
What types of people do you have to help manage your clinical
knowledge?
Do you have specific individuals responsible for individual pieces of
clinical content?
Does each piece of clinical content (i.e., alert, order set, and patient
education material) have an individual responsible for maintaining it?
Automatic reminders that content review date is approaching sent to
individuals responsible for maintaining the content
Automatic notifications when clinical guidelines underlying CDS change
System in place to “learn” new clinical knowledge from your patient
database
Do you purchase, or share, clinical knowledge from external sources…
  Reference information
Drug information (dose ranges, interactions, patient handouts,
indications, etc.)
    Order sets
    Alerts / Reminders

Footnotes

Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.

Conflict of interest None of the authors reports any conflict of interest.

References

1. Kohn L, Corrigan J, Donaldson MS. Institute Of Medicine. The National Academy Press; Washington, DC: 2000. To Err Is Human: Building a Safer Health System.
2. Committee on Quality of Healthcare in America . Institute Of Medicine. The National Academy Press; Washington, D.C.: 2001. Crossing the Quality Chasm: A New Health System for the 21st Century. [PubMed]
3. Linder JA, Ma J, Bates DW, Middleton B, Stafford RS. Electronic health record use and the quality of ambulatory care in the United States. Arch Intern Med. 2007 Jul 9;167(13):1400–5. [PubMed]
4. Osheroff JA, Pifer EA, Teich JM, Sittig DF, Jenders RA. Improving Outcomes with Clinical Decision Support: An Implementer’s Guide. Health Information Management and Systems Society; 2005.
5. Rocha RA, Bradshaw RL, Bigelow SM, et al. Towards ubiquitous peer review strategies to sustain and enhance a clinical knowledge management framework. AMIA Annu Symp Proc. 2006:654–8. [PMC free article] [PubMed]
6. Shiffman RN. Guideline maintenance and revision. 50 years of the Jones criteria for diagnosis of rheumatic fever. Arch Pediatr Adolesc Med. 1995 Jul;149(7):727–32. [PubMed]
7. Sittig DF, Wright A, Osheroff JA, Middleton B, Teich JM, Ash JS, Campbell E, Bates DW. Grand challenges in clinical decision support. J Biomed Inform. 2008;41(2):387–92. [PMC free article] [PubMed]
8. Bali RK, editor. Clinical Knowledge Management: Opportunities and Challenges. Idea Group Publishing; 2005.
11. Chaudhry B, Wang J, Wu S, Maglione M, Mojica W, Roth E, Morton SC, Shekelle PG. Systematic review: impact of health information technology on quality, efficiency, and costs of medical care. Ann Intern Med. 2006 May 16;144(10):742–52. [PubMed]
12. Davies Organizational Award Recipient Manuscripts. (available at: http://www.himss.org/ASP/davies_organizational.asp.
13. The Joint Commission’s Venous Thromboembolism (VTE) Core Measure Set. [accessed 3 June 2009]. http://www.jointcommission.org/PerformanceMeasurement/PerformanceMeasurement/VTE.htm.
14. Wright A, Bates DW, Middleton B, Hongsermeier T, Kashyap V, Thomas SM, Sittig DF. Creating and Sharing Clinical Decision Support Content with Web 2.0: Issues and Examples. Journal of Biomedical Informatics. 2009 Apr;42(2):334–46. [PubMed]
15. Cimino JJ. Terminology tools: state of the art and practical lessons. Methods Inf Med. 2001;40(4):298–306. [PubMed]
16. Baorto D, Li L, Cimino JJ. Practical experience with the maintenance and auditing of a large medical ontology. J Biomed Inform. 2009 Jun;42(3):494–503. [PMC free article] [PubMed]
17. Chute CG, Elkin PL, Sherertz DD, Tuttle MS. Desiderata for a clinical terminology server. Proc AMIA Symp. 1999:42–6. [PMC free article] [PubMed]
18. Dolin RH, Mattison JE, Cohn S, et al. Kaiser Permanente’s Convergent Medical Terminology. Stud Health Technol Inform. 2004;107(Pt 1):346–50. [PubMed]
19. Apelon DTS 3.4.1. available at: http://sourceforge.net/forum/forum.php?forum_id=799582.
20. Shah NR, Seger AC, Seger DL, Fiskio JM, Kuperman GJ, Blumenfeld B, Recklet EG, Bates DW, Gandhi TK. Improving acceptance of computerized prescribing alerts in ambulatory care. J Am Med Inform Assoc. 2006 Jan-Feb;13(1):5–11. [PMC free article] [PubMed]
22. Wright A, Sittig DF. A framework and model for evaluating clinical decision support architectures. J Biomed Inform. 2008 Mar 25; [PMC free article] [PubMed]
23. McDonald CJ. Protocol-based computer reminders, the quality of care and the non-perfectability of man. N Engl J Med. 1976 Dec 9;295(24):1351–5. [PubMed]
24. Wright A, Sittig DF. Automated development of order sets and corollary orders by data mining in an ambulatory computerized physician order entry system. AMIA Annu Symp Proc. 2006:819–23. [PMC free article] [PubMed]
25. Sittig DF, Wright A, Osheroff JA, Middleton B, Teich JM, Ash JS, Campbell E, Bates DW. Grand challenges in clinical decision support. J Biomed Inform. 2008 Apr;41(2):387–92. [PMC free article] [PubMed]
26. Agency for Healthcare Research and Quality (AHRQ) -- Clinical Decision Support Initiative. http://healthit.ahrq.gov/portal/server.pt?open=512&objID=654&&PageID=13665&mode=2&in_hi_userid=3882&cached=true.
27. The Joint Commission’s CORE Measure Initiative. Available at: http://www.jointcommission.org/PerformanceMeasurement.
28. The Joint Commission’s National Patient Safety Goals Initiative. Available at: http://www.jointcommission.org/GeneralPublic/NPSG/09_npsgs.htm.
29. Medicare’s Physician Quality Reporting Initiative. Available at: http://www.cms.hhs.gov/PQRI/01_Overview.asp.
30. Alavi M, Leidner DE. Knowledge Management and Knowledge Management Systems: Conceptual Foundations and Research Issues. MIS Quarterly. 2001;25(1):107–136.
31. Damodaran L, Olphert W. Barriers and facilitators to the use of knowledge management systems. Behaviour & Information Technology. 2000;19(6):405–413.
32. Hlupic V, Pouloudi A, Rzevski G. Towards an Integrated Approach to Knowledge Management: ‘Hard’, ‘Soft’ and ‘Abstract’ Issues. Knowledge and Process Management. 2002;9(2):90–102.
33. Chin HL, et al. The comprehensive computer-based patient record in Kaiser Permanente Northwest. In: Overhage JM, editor. Proc Fourth Ann Davies Award. McGraw-Hill; New York: 1998. pp. 69–129.
34. Jensen J. The effects of Computerized Provider Order Entry on medication turn-around time: a time-to-first dose study at the Providence Portland Medical Center. AMIA Annu Symp Proc. 2006:384–8. [PMC free article] [PubMed]
35. McDonald CJ, Tierney WM. The Medical Gopher--a microcomputer system to help find, organize and decide about patient data. West J Med. 1986 Dec;145(6):823–9. [PMC free article] [PubMed]
36. Teich JM, Spurr CD, Flammini SJ, et al. Response to a trial of physician-based inpatient order entry. Proc Annu Symp Comput Appl Med Care. 1993:316–20. [PMC free article] [PubMed]
37. Department of Veterans Affairs. Office of Information . VistA Monograph, 2005-2006. Department of Veterans Affairs, Office of Information; May, 2005. Available at http://www.va.gov/vista_monograph/docs/vista_monograph2005_06.pdf.
38. Case Study: Rapid Development of Order Sets for CPOE Go-Live at Memorial Hermann. http://www.zynxhealth.com/newsletters/EyeOnZynx/20080131/Features/CaseStudy/MemorialHermann_print.htm.
39. Kuperman GJ, Fiskio JM, Karson A. A process to maintain the quality of a computerized knowledge base. Proc AMIA Symp. 1999:87–91. [PMC free article] [PubMed]
40. Geissbuhler A, Miller RA. Distributing knowledge maintenance for clinical decision-support systems: the “knowledge library” model. Proc AMIA Symp. 1999:770–4. [PMC free article] [PubMed]
41. Shah NR, Seger AC, Seger DL, Fiskio JM, Kuperman GJ, Blumenfeld B, Recklet EG, Bates DW, Gandhi TK. Improving acceptance of computerized prescribing alerts in ambulatory care. J Am Med Inform Assoc. 2006 Jan-Feb;13(1):5–11. [PMC free article] [PubMed]
43. De Clercq P, Kaiser K, Hasman A. Computer-Interpretable Guideline formalisms. Stud Health Technol Inform. 2008;139:22–43. [PMC free article] [PubMed]
44. Wright A, Sittig DF. A framework and model for evaluating clinical decision support architectures. J Biomed Inform. 2008 Dec;41(6):982–90. [PMC free article] [PubMed]
45. Powsner SM, Riely CA, Barwick KW, Morrow JS, Miller PL. Automated bibliographic retrieval based on current topics in hepatology: Hepatopix. Comput Biomed Res. 1989;22:552–64. [PubMed]
46. Powsner SM, Miller PL. Automated online transition from the medical record to the psychiatric literature. Methods Inf Med. 1992;31:169–74. [PubMed]
47. Cimino JJ, Elhanan G, Zeng Q. Supporting infobuttons with terminological knowledge. Proc AMIA Annu Fall Symp. 1997:528–32. [PMC free article] [PubMed]
48. Maviglia SM, Yoon CS, Bates DW, Kuperman G. KnowledgeLink: impact of context-sensitive information retrieval on clinicians’ information needs. J Am Med Inform Assoc. 2006 Jan-Feb;13(1):67–73. [PMC free article] [PubMed]
49. Osheroff JA, editor. Improving Medication Use and Outcomes with Clinical Decision Support A Step-by-Step Guide. Healthcare Information and Management Systems Society; 2009.
50. Ash JS, Stavri PZ, Kuperman GJ. A consensus statement on considerations for a successful CPOE implementation. J Am Med Inform Assoc. 2003 May-Jun;10(3):229–34. [PMC free article] [PubMed]
51. Goldstein MK, Coleman RW, Tu SW, Shankar RD, O’Connor MJ, Musen MA, Martins SB, Lavori PW, Shlipak MG, Oddone E, Advani AA, Gholami P, Hoffman BB. Translating research into practice: organizational issues in implementing automated decision support for hypertension in three medical centers. J Am Med Inform Assoc. 2004 Sep-Oct;11(5):368–76. [PMC free article] [PubMed]
52. Shiffman RN. Guideline maintenance and revision. 50 years of the Jones criteria for diagnosis of rheumatic fever. Arch Pediatr Adolesc Med. 1995 Jul;149(7):727–32. [PubMed]
53. Sittig DF, Wright A, Osheroff JA, Middleton B, Teich JM, Ash JS, Campbell E, Bates DW. Grand challenges in clinical decision support. J Biomed Inform. 2008 Apr;41(2):387–92. [PMC free article] [PubMed]
54. Kuperman GJ, Reichley RM, Bailey TC. Using commercial knowledge bases for clinical decision support: opportunities, hurdles, and recommendations. J Am Med Inform Assoc. 2006 Jul-Aug;13(4):369–71. [PMC free article] [PubMed]
55. Dolin RH, Mattison JE, Cohn S, et al. Kaiser Permanente’s Convergent Medical Terminology. Medinfo. 2004;11(Pt 1):346–50. [PubMed]
57. EMC Documentum eRoom – Extending the power and value of team collaboration. EMC Corp.; 2006.