Overview of organizations
These five organizations encompass a variety of academic and community hospitals, and comparatively small and large ambulatory practices. They use both commercially available and in-house-developed clinical information systems. They also include for-profit, not-for-profit, and government-provided healthcare as well as the full gamut of medical and surgical clinical specialties. Their key characteristics are shown in . We visited Partners, MVIPA, and the VA Health System in Indianapolis, Indiana. The University of Texas and Vanderbilt were used as our validation sites and were characterized through written questions and interviews.
Overview of the organizations' characteristics
In addition to the broad demographic characteristics in , we found that each site had at least one example of each of the six decision-support types in the Osheroff taxonomy, suggesting that each site had sufficient breadth of content to necessitate some level of decision-support governance.
Despite having similar breadth of CDS, the sites varied dramatically in the depth of decision support they had. For example, the number of different condition-specific order sets ranged from less than 50 to more than 800, and the number of unique clinical alerts ranged from less than 50 to more than 7000. This pattern of differing “depths” of CDS implementation accounts in large part for the differences in the structure, complexity, and size of the clinical governance infrastructure that each organization has developed.
All five sites in our analysis had at least some degree of decision-support governance, although the form and pattern of governance models differed greatly. describes the approaches, including CDS-related staff, committees, governance process, tailoring of content, levels of governance, and tools for soliciting and managing user feedback. The five organizations are listed left to right across the top according to size, starting with the two smaller strictly ambulatory sites that use commercial systems. They are followed by the two academic health centers with locally developed inpatient as well as outpatient systems. The national VA health network, in the final column, is the largest and includes inpatient, outpatient, long term, and home-based care. compares governance styles and structures across each of the five case studies. See appendix 1, available as an online data supplement (http://www.jamia.org
), for further detail on the case studies.
Governance approaches at each of the five sites
Variation in governance strategies was observed across the five sites. CDS-related staff ranged from a small staff responsible for overseeing a vendor system to large academic informatics departments that included numerous clinicians, informaticians, and software developers. Most sites employed the use of committees to govern development, implementation, and maintenance of CDS systems, although committee structure and overall organization varied. In some cases, content was purchased from an outside vendor, and in others it was developed by in-house staff based on institutional needs. Sites also varied in the degree to which they allowed tailoring of CDS content in individual practices or hospital services. Some form of regular review was common across all sites studied, although mechanisms of user feedback varied significantly in form and frequency.
One particularly notable finding was that governance approaches differed substantially between institutions relying on vendor-based clinical information systems and those with “home-grown” internal clinical information systems (CIS). These differences were largely related to each type of institution's involvement in CDS development, implementation and management. Organizations that rely on vendor-based CDS require governance structures largely limited to collaboration with vendors and ongoing day-to-day operational management. In contrast, organizations with internally developed systems are responsible for the entire process of development, implementation, and ongoing management and assessment. These fundamental differences result in diverging governance practices, which are summarized in .
Comparison of governance practices at sites with internal CIS (clinical information systems) development and vendor-based CIS
Overview of content-management approaches
Our research findings clearly revealed that sites needed coordinated ways to manage their decision-support content, particularly as the amount and complexity of the content grew.
At Partners, a great deal of the CDS content was developed as part of research projects, and balancing research and operational decision-support projects has been challenging, as research content must be maintained (or decommissioned) after the research project ends. Over the last 5 years, with the maturation of the knowledge management group's processes, content is increasingly re-evaluated on a periodic basis, which includes analysis of usage data, such as acceptance and override rates for alerts.22
The most robust review mechanisms are in place for formulary drug information (dose and frequency lists), drug–drug interaction rules,24
age- and renal-based drug dosing guidance rules,25
and health maintenance reminders. For these assets, Partners conducts regularly scheduled reviews of content, at differing frequencies. In addition, on-demand reviews are prompted by user feedback and external events gleaned from regular scans of health-information sources, such as FDA and pharmaceutical company announcements and the medical research literature.
Vanderbilt has several mechanisms to assess ongoing proper functioning of clinical content. Surveillance data are collected on order set usage, responses to decision-support pop-up alerts, and other CDS items to adjust the support to the desired outcome. Vanderbilt conducts regular reviews of content and then updates the content as clinical evidence surfaces that would necessitate a change. Vanderbilt prioritizes changes based on quality, safety, clinical volumes, and cost/benefit.
In contrast to the other academic institutions, the University of Texas, Houston Physicians' Practice Plan relies heavily on Allscripts' content, and updates are tied to the Allscripts' release schedule. The Practice Plan conducts surveys of clinical users and creates monthly reports that provide all clinics with a breakdown of the utilization of all aspects of the EHR (electronic health record) (eg, percentage of prescriptions written electronically).
CDS capabilities at the VA are constantly being updated and revised to reflect internal research activities, review of performance measures, input from regulatory groups, networking with academic affiliates, changing emphasis of national program and patient priorities, and involvement in national professional societies and working groups. Evidence-based guidelines from nationally recognized authorities are also reviewed regularly for inclusion. Additionally, the VA national pharmacy program is pursuing integration of a commercially available database for drug–drug/allergy interactions, which would be maintained and periodically updated. Evaluation of the effectiveness of the CDS is indirect and based on the measurement of outcomes it was designed to impact.
Finally, at MVIPA, the primary method of evaluating the effectiveness of CDS is ongoing field testing. Content is updated periodically through the NextGen release cycle and is first tested by MVIPA staff. Design changes that affect clinical content are reviewed by the medical director of information systems. The director also monitors regulatory requirements and prioritizes updates according to their effective dates and their impact.
Comparison of content-management approaches
The content-management practices of the organizations differed significantly based on the nature of the organizations. Academic medical centers reported that much of their content was, at least initially, developed by researchers as a part of research projects. Identifying and maintaining this content, particularly after research projects concluded, appeared to be challenging. Sites also reported developing content based on user requests, regulatory and quality reporting requirements, and other organizational priorities.
All sites reported some form of ongoing content review, but the frequency and regularity of these reviews varied. Some organizations carried out an annual review, while other organizations reviewed content based on user feedback or as new evidence became available. Sites which relied more heavily on vendor content were more likely to report periodic updates or refreshes, often tied to vendor release schedules.
Sites also mentioned and underscored the importance of field-testing content and listening and responding to user feedback. These feedback mechanisms ranged from passive receipt of reports to active feedback tools embedded in the EHR and periodic site visits and assessments. Some sites also reported monitoring usage of content in order to prioritize the most highly used content.
In addition to direct CDS-related evaluation, some sites (particularly the VA) reported a strong focus on associated quality measures. The VA also emphasizes standard practice and common data elements within and across their system of care, and is in the process of standardizing such practices when there is evidence or experience to support such efforts. Other sites, including Partners, reported less standardization.