Clinical decision support: progress and opportunities

doi:10.1136/jamia.2010.005561

J Am Med Inform Assoc. 2010 Sep-Oct; 17(5): 487–492.

doi: 10.1136/jamia.2010.005561

PMCID: PMC2995690

Clinical decision support: progress and opportunities

Jason A Lyman,¹ Wendy F Cohn,¹ Meryl Bloomrosen,² and Don E Detmer^1,²

¹Department of Public Health Sciences, University of Virginia Health System, Charlottesville, Virginia, USA

²American Medical Informatics Association, Bethesda, Maryland, USA

Corresponding author.

Correspondence to Dr Jason Lyman, Department of Public Health Sciences, University of Virginia Health System, Charlottesville, VA 22908, USA; Email: lyman/at/virginia.edu

Any opinions, findings, conclusions, or recommendations expressed in this publication are those of the author(s) and do not necessarily reflect the views of AMIA or the Commonwealth Fund. Statements in the report should not be construed as endorsement by AMIA or the Commonwealth Fund.

Received April 29, 2009; Accepted June 25, 2010.

This article has been cited by other articles in PMC.

Abstract

In 2005, the American Medical Informatics Association undertook a set of activities relating to clinical decision support (CDS), with support from the office of the national coordinator and the Agency for Healthcare Research and Quality. They culminated in the release of the roadmap for national action on CDS in 2006. This article assesses progress toward the short-term goals within the roadmap, and recommends activities to continue to improve CDS adoption throughout the United States. The report finds that considerable progress has been made in the past four years, although significant work remains. Healthcare quality organizations are increasingly recognizing the role of health information technology in improving care, multi-site CDS demonstration projects are under way, and there are growing incentives for adoption. Specific recommendations include: (1) designating a national entity to coordinate CDS work and collaboration; (2) developing approaches to monitor and track CDS adoption and use; (3) defining and funding a CDS research agenda; and (4) updating the CDS ‘critical path’.

The quality and safety of medical care in the United States have drawn increased attention in the past decade. Studies suggest many errors could be avoided with the use of health information and communications technology (HIT).ⁱ ^1–4 Such improvements have been facilitated by the adoption of computerized provider order entry systems, electronic medical records that improve accessibility to clinical data, and a variety of approaches loosely grouped together and referred to as clinical decision support (CDS) systems. To foster better health processes, better individual patient care, and better population health, CDS systems intelligently provide, at appropriate times, knowledge or information (person-specific or population-specific). Clinicians, patients and individuals thus benefit from CDS.⁵ Clinical decision support interventions may include alerting and reminder systems, dosing calculators, and order sets and tools that provide access to medical knowledge at the point of care. Evidence suggests that computerization of medical record systems and even implementation of provider order entry systems may not be sufficient to ensure high quality care.⁶ Rather, CDS represents the effecter arm for clinical process improvement,^2–4 provided that it is effectively utilized and implemented with careful consideration of clinical workflow.

In the summer of 2005, the Office of the National Coordinator for Health Information Technology (ONC), along with the Agency for Healthcare Research and Quality (AHRQ) asked the American Medical Informatics Association (AMIA) to develop a plan to guide federal and private sector activities to advance CDS. In response, AMIA established the CDS roadmap development steering committee to lead this effort. A set of meetings and consensus panels led to the production of the roadmap for national action on CDS (the ‘CDS roadmap’) in 2006.⁵ This report recommended activities to facilitate CDS development, implementation and use throughout the United States to improve the quality, safety and efficiency of healthcare. The roadmap included a critical path that recommended activities in the three-year timeframe following the report's publication.

Since then, significant effort by numerous stakeholders, including federal agencies, quality organizations, informatics groups, healthcare systems and individual researchers have devoted effort to CDS. To assess national progress in CDS, we conducted an environmental scan, reviewing published literature, white papers, reports by multiple stakeholders and recent legislation. Using the critical path activities as a framework, our report presents a synthesis of progress to date. We discuss future directions and recommend specific next steps, taking into consideration trends in clinical computing and increased availability of funds to support HIT as part of the recent US federal stimulus package.

The CDS roadmap and the critical path

The CDS roadmap organizes its recommendations into three pillars (‘best knowledge available when needed’, ‘high adoption and effective use’ and ‘continuous improvement of knowledge and CDS methods’), with each pillar subdivided into two strategic objectives (table 1).⁵ A comprehensive work plan in the roadmap suggests a detailed list of actions across a broad timeline.

Table 1

CDS roadmap pillars and strategic objectives

The roadmap also lays out a set of short-term critical path activities, focused on the three-year time horizon from 2006 to 2009. Suggestions include an executive steering group to coordinate and facilitate progress, and efforts to share knowledge using implementation guides and CDS starter sets. The critical path also recommended funding demonstration projects to establish the feasibility of CDS dissemination beyond benchmark organizations and identify best practices that could help facilitate broader CDS adoption. In particular, the critical path included the following eight items, which we have grouped into five categories for ease of analysis and discussion (table 2). Note that the category ‘coordination and oversight’ has been applied to both the first and last critical path activity. In the narrative that follows, we summarize key progress in each of these five categories.

Table 2

Critical path goals

CDS coordination and oversight

According to the roadmap authors, one limiting factor in the efficient use of CDS was the lack of a single coordinating entity that could oversee strategic development and deployment of these tools. To address this need, the authors suggested the formation of the roadmap execution steering group. While such a group has not been formally chartered, similarly focused groups and committees have formed. At the federal level, three entities are seeking to catalog or coordinate CDS activities: the CDS collaboratory,⁷ AHRQ,⁸ and the Department of Health and Human Services personalized healthcare initiative.⁹ Similarly, the American Health Information Community (AHIC), a federal advisory committee chartered at the behest of the Department of Health and Human Services, convened an ad hoc CDS planning group in 2007, and in the spring of 2008 developed a set of recommendations for federal action. They included the development of a national consensus statement on priorities for CDS interventions, greater federal funding for CDS, and the development of a public–private partnership with broad stakeholder involvement to help coordinate specific CDS activities and facilitate dialog and collaboration. At the end of 2008, AHIC ceased its operations and transitioned into a public–private entity, the National eHealth Collaborative. Since then, the fate of the planning group's recommendations remains unclear, perhaps one consequence of the continued absence of a single designated coordinating body for CDS development and deployment.

CDS and quality improvement

The authors of the roadmap highlighted the importance of alignment between healthcare quality organizations and CDS initiatives. Such organizations, including AHRQ and the National Quality Forum (NQF), are increasingly recognizing the potential benefit of CDS. At the end of 2008, the national priorities partnership, a group of 28 major organizations convened by the NQF, identified a set of national priorities and goals for the US healthcare system to help guide performance improvement programs. They explicitly mentioned computerized provider order entry (CPOE) and CDS as useful tools for improving safety by reducing errors.¹⁰ The NQF has also developed national voluntary consensus standards for health information technology: structural measures (2008), which focuses on the assessment of the extent to which tools like e-prescribing, electronic health records (EHR), CDS systems and CPOE are being utilized. The Leapfrog Group continues as an active promoter of the role of HIT, including CDS, in the improvement of the quality and safety of health care. One of its initiatives, the hospital rewards program, offers incentives for hospitals that perform well in an annual hospital survey that measures, in part, CPOE adoption and utilization. The Leapfrog Group is also launching a CPOE executive consortium, with an anticipated focus on the development and dissemination of best practices in adoption and implementation, research on CPOE effectiveness, and a starter set of CDS rules. As evidenced by these efforts, healthcare quality organizations are increasingly placing value on the potential role of decision support in improving clinical performance.

Knowledge sharing

Roadmap authors stressed the importance of sharing CDS knowledge and lessons learned among organizations, emphasizing the role of dissemination efforts, EHR certification requirements and the development of CDS starter sets. Significant progress has been made to meet this objective. A recent publication, edited by JA Osheroff, provides a step-by-step guide to improving medication use and outcomes with CDS; it was co-published by AMIA, the American Society of Health-System Pharmacists, the Association of Medical Directors of Information Systems, Healthcare Information and Management Systems Society, the Institute For Safe Medication Practices and the Scottsdale Institute, with further sponsorship by other organizations.¹¹ RA Greenes recently published a textbook devoted to CDS, which addresses the topic from a practical as well as a scholarly perspective.¹² The AHRQ has created a national resource center for health information technology, with a goal of describing and disseminating findings arising from the multiple grants it has funded in this area. Numerous resources that pertain to CDS are available. AHRQ also conducts frequent webinars on a variety of topics related to CDS, including toolkits, evaluation approaches and workflow considerations.⁸ They recently published a report describing lessons learned from two large AHRQ-funded CDS demonstration projects highlighting challenges faced at the study sites and recommendations for addressing them.¹³

Knowledge sharing is also a major focus of the Morningside Initiative, a collaborative, multidisciplinary group whose members include AMIA, Arizona State University, the Department of Defense (Military Health System), the Henry Ford Health System, Intermountain Health Care, Kaiser Permanente, Partners Healthcare, Telemedicine & Advanced Technology Research and the Veterans Healthcare Administration. This group seeks to enable broader adoption of CDS by focusing on collaborative knowledge management. One key initiative for this organization is a shared CDS knowledge repository. The Clinical Decision Support Consortium (CDSC), based at Partners Healthcare in Boston, is an AHRQ-funded initiative that also devotes considerable attention to knowledge sharing (described in greater detail below).

A recent trend is the use of emerging technologies, including social networking tools, to facilitate grassroots collaboration. A Wright and colleagues¹⁴ have summarized several such approaches, including the ClinfoWiki, a wiki site devoted to informatics with a large section focused on CDS. The vendor community appears to be following suit. Of note is Epic Systems Corporation, developer of the EpicCare EMR system, that hosts a community library on its site to facilitate knowledge sharing among its clients; it provides access to CDS tools, such as order sets, documentation templates and rule-based alerts developed at other sites.

Demonstration projects

The critical path also included the development, funding and implementation of CDS demonstration projects as one of the core actions to establish the feasibility of CDS dissemination beyond benchmark organizations and facilitate broader CDS adoption. In the past three years, AHRQ has funded several such initiatives, including CDS-related grants, contracts and demonstration projects.¹³ ¹⁵

Guidelines into Decision Support

The Guidelines into Decision Support (GLIDES) project, a collaborative venture between the Yale School of Medicine, Yale New Haven Health and Nemours Health System, seeks to integrate evidence-based guidelines for pediatric obesity and asthma into ambulatory care practices, which use EHR products by GE and Epic. This two-year project, which was completed in early 2010, utilizes the guidelines encoding model and associated tools for translating guidelines into computer-readable formats and supporting their implementation and maintenance.¹³ ¹⁶ ¹⁷

Clinical Decision Support Consortium

The Clinical Decision Support Consortium (CDSC) is another AHRQ-funded initiative that includes healthcare provider organizations, commercial entities and other groups. Clinical domains include diabetes mellitus, coronary artery disease and hypertension. Work is organized into 10 teams that cover a variety of critical areas including the knowledge management lifecycle, CDS services, evaluation and dissemination of best practices. A specific research objective deals with creating re-usable knowledge content and enabling dissemination through the use of web services.¹³ ¹⁸

Centers for Education and Research on Therapeutics

The mission of Centers for Education and Research on Therapeutics (CERT) is to conduct research and provide education for several purposes. They include advancing the optimal use of drugs, medical devices and biological products, increasing awareness of the benefits and risks of therapeutics, and improving quality while cutting costs of care. Brigham and Women's Hospital has a center that deals specifically with health information technology, and several projects under way relate to CDS.¹⁹

Legal, regulatory and financial issues

The critical path included steps to address a variety of external challenges related to the adoption and use of CDS, including financial incentives, statutory requirements and federal and state regulations. Notable events in this category include the passage of comprehensive federal legislation, the release of draft regulations addressing the utilization of health information technology and emerging trends related to the business case for CDS.

The economic climate and stimulus package

The US economy has recently undergone a dramatic downturn on a scale not seen in decades. The impact this will have on the rate of HIT adoption is unclear; some evidence suggests that delays due to economic distress may occur. A survey by the American Hospital Association in December 2008 showed that almost 62% of responding hospitals planned to delay information technology projects.²⁰ The NCR's 2009 survey of 435 hospitals found that 71% expected smaller budgets for information technology in the upcoming fiscal year.²¹ However, both surveys were conducted before the passage of the 2009 American Recovery and Reinvestment Act that provides a total potential payout of $34 billion in infrastructure and centers for medicare and medicaid services (CMS) incentives to promote HIT adoption.²² Such resources will likely support implementation of the 2008 ONC-coordinated federal health information technology strategic plan, a five-year plan to improve both patient-focused health care and population health through the use of HIT.²³ Key objectives of the plan directly relate to the necessary foundation for CDS adoption, including privacy, confidentiality and security, interoperability, EHR adoption and collaborative governance. The plan sets an EHR adoption target of 40% of physician offices by 2012 and enumerates multiple strategies to reduce barriers and to provide practice incentives to help meet this goal.

Meaningful use

The American Recovery and Reinvestment Act also includes incentives to address clinician use of HIT. Eligible professionals and provider organizations who meet specific criteria can expect increased reimbursement from CMS. Over time, these incentives are planned to evolve into penalties for those who do not meet criteria. The final days of 2009 saw the release of two proposed CMS regulations that have the potential to dramatically impact the future of CDS.ⁱⁱ A notice of proposed rule-making includes criteria for bonus payments to providers and hospitals.²⁴ Some criteria directly address CDS, and others facilitate its deployment by focusing on electronic capture of underlying clinical data. Examples include drug–drug, drug–allergy and drug–formulary checks, and the implementation of CDS rules targeting high priority conditions. Eligible providers and hospitals must track compliance with the alerts triggered by these rules. Also included are requirements for CPOE, the use of evidence-based order sets, e-prescribing and patient reminders for preventive health testing. The second regulation released in December 2009 is an interim final rule that specifies certification criteria for EHR.²⁵ This regulation states the qualifying EHR must include CDS rules that use ‘demographic data, specific patient diagnoses, conditions, diagnostic test results and/or patient medication list’. In addition, these systems must ‘automatically and electronically generate and indicate (eg, pop-up message or sound) in real-time, alerts and care suggestions based upon CDS rules and evidence grade’. User responses to such alerts must be tracked as well as used to generate reports for monitoring purposes.

The business case

As noted by roadmap authors, the disparity between who pays for the investment of CDS development and deployment and who benefits creates barriers for adoption. Cost savings accrue to healthcare payers, although it is almost always provider organizations who purchase (or develop) these tools. A recent initiative in the area of diagnostic imaging, however, employs a different strategy. A pilot project involving the Institute for Clinical Systems Improvement, the Minnesota Department of Human Services and the collaboration of six medical groups and five health plans demonstrated that the use of a decision support system for imaging orders improved the appropriateness of such orders. As a result, the health plans purchased the decision support tool for use by participating provider organizations. It remains to be seen whether this model will expand to other geographical areas or specific types of CDS modules.

Summary and suggested next steps

Three years ago, as part of an overall framework for CDS advancement in the USA, a group of experts identified a set of tasks to help support broader use of these tools. Significant progress has been made in most of the areas described, but much work is still needed. Demonstration projects in CDS have been funded and are under way. The healthcare quality community increasingly recognizes the role of HIT and supports greater use of CDS. Commercial entities, federal organizations, non-profit groups and academic organizations are all contributing approaches to help disseminate important knowledge about CDS deployment. Incentives for provider adoption of these tools are on the rise. We believe that continued progress in this area would be greatly facilitated by four key activities: (1) the designation of a single entity to coordinate CDS development and deployment; (2) the development and use of techniques to measure the use of CDS tools; (3) continued research funding for CDS; and (4) the development of an updated critical path.

Designation of an entity to coordinate CDS work and collaboration

Although coordination and oversight of CDS was a theme in both the roadmap and the recommendations provided by the AHIC ad hoc CDS working group, no single oversight group has emerged. To be effective, such a group must include representatives from quality organizations, healthcare providers, payers, regulators and vendors. Two important areas that could be potentially addressed by a coordinating entity include CDS knowledge management and the promotion of a common terminology.

Supporting knowledge management

Order sets, alerts, templates and other CDS tools have a lifecycle that requires underlying organizational and technological infrastructure. Organizations must have processes in place to handle requests for new CDS interventions, including prioritization and vetting, as well as mechanisms for tracking the performance of existing interventions. In addition, interventions rely on clinical knowledge that is constantly changing and so must be regularly reviewed and updated. Knowledge management is a major focus of the CDSC initiative described earlier. While informaticians are likely familiar with these requirements, it may take effort to convince leadership within provider organizations that resources must be provided to support these activities. A coordinating group might make specific recommendations in this area and begin to develop standard approaches to this common but underrecognized need.

Promotion of a common terminology

A possible benefit of such collaboration could be the development of a common terminology around CDS. The current inconsistency of terms and resulting lack of clarity was specifically mentioned in the CMS notice of proposed rule-making described earlier, and highlighted in a CDS town hall meeting at the 2008 AMIA fall symposium.²⁶ To some, CDS indicates alerts and reminders using rule-based algorithms. To others, the term refers to a broader suite of tools and approaches, including the use of evidence-based order sets, documentation templates that prompt the collection of desired information, and intelligent display of information at the time it is needed. The lack of agreed-upon definitions can impede thoughtful discussion and decision-making about CDS adoption and implementation, with oft-used terms sometimes reflecting the emotional response that CDS can generate. Terms like ‘alert fatigue’, ‘hard stop’ and ‘pop-up’ refer to the interruptive nature of some CDS tools, and indiscriminate use of these phrases may impede thoughtful discussion of how HIT can be leveraged. The vendor community has a critical role to play in this regard. Provider organizations are likely to adopt the terminology used by the EHR developer to facilitate clear communication among designers, developers and decision-makers during and after implementation.

Several organizations appear to be well positioned to take on a coordinating role. The ONC could designate a specific workgroup to oversee CDS activities. The eHealth Initiative, a non-profit organization with members that include healthcare quality organizations, professional organizations, payers, consumer groups, healthcare provider organizations, public health agencies and others, may also be well suited for this role.

Develop approaches to monitor and track CDS adoption and use

A system for measuring the rate of adoption and utilization of CDS tools would provide valuable feedback on the effectiveness of the many incentive programs and other initiatives aimed at furthering deployment of these technologies. The structural measures developed by the NQF, evaluation measures available through the AHRQ, and metrics used by Leapfrog provide a solid foundation for developing a system to track CDS deployment. The creation of a standard instrument that could be used to measure CDS utilization could help us assess trends in the national adoption of these technologies.

Define and fund a CDS research agenda

Significant gaps remain in our understanding of how to optimize HIT for effective clinical decision-making. Of the 10 ‘grand challenges’ described by Sittig et al,²⁷ many can only be successfully addressed with innovative thinking and careful, academic study. A few of the cited challenges that continue to require investigation are: improving the human–computer interface in the context of clinical workflow, creating better summaries of patient-level information, learning how to best prioritize and filter recommendation, and effectively applying natural language processing to free text to identify the ‘triggering’ conditions necessary for CDS. In a recent publication edited by Stead et al,²⁸ the authors define several critical areas that require study, such as the need for a better understanding of how to effectively deliver ‘patient-oriented cognitive support.’ Their recommendations include the need for more interdisciplinary research on systems-based approaches to healthcare process improvement and clinically oriented human–computer interaction.

As a result of the federal economic recovery program, the National Institutes of Health recently released a request for applications targeting specific ‘challenge areas’ in health and science research including several topics related to HIT.²⁹ One topic in particular, ‘advanced decision support for complex clinical decisions’, explicitly focuses on the need for innovative approaches to support clinical decision-making. The AHRQ has also issued a set of program announcements focused on the use of HIT to improve quality.³⁰ ³¹ Finally, the health information technology for economic and clinical health priority grants program recently announced the strategic health IT advanced research projects program that includes patient-centered cognitive support as one of four topic areas.³² Such funding opportunities are needed to address fundamental limitations in our knowledge about CDS. Increased funding for the National Library of Medicine to support grants in this area would also be greatly beneficial, given that agency's significant role in supporting applied informatics research and development.

Update the critical path

The critical path in the roadmap laid out a set of short-term objectives, many of which have been addressed. Although the extent to which the critical path spurred these developments is unclear, an updated critical path that describes a clear set of activities for the next three years may well support further progress in a coordinated manner. Such a planning document could take advantage of the ONC coordinated federal health information technology strategic plan, recommendations of the AHIC ad hoc CDS planning group and new CMS regulations on EHR certification and meaningful use; it might ease the transition of providers, clinics and hospitals toward more efficient use of HIT including CDS. This new critical path could be organized by the coordinating body for CDS recommended above.

Conclusion

The amount of energy and resources that have been devoted to CDS planning efforts in recent years is considerable and impressive. The availability of significant federal resources to support these efforts has the potential to strengthen and expedite these efforts. Many challenges remain, but we expect that reimbursement strategies will continue to align with quality initiatives in areas that CDS has the potential to address, and new knowledge will emerge from the AHRQ-funded demonstration projects and other research activities. We believe that these challenges can and will be overcome, particularly if coordination of efforts across organizations working in this arena can be improved.

Acknowledgments

The authors wish to express their thanks to Freda Temple, G Pope Atkins and Joan Atkins for their careful review and editing of this manuscript.

Footnotes

Funding: This paper was supported by grant no 20070835, between the American Medical Informatics Association (AMIA) and the Commonwealth Fund.

Ethics approval: This study was conducted with the approval of the N/A.

Provenance and peer review: Not commissioned; externally peer reviewed.

ⁱFor purposes of clarity, we are using the term health information technology (HIT) to refer to health information and communications technology. It is worth noting that in much of the world, this domain is typically referred to as health information and communication technology (HICT), which we feel is a more apt approach.

ⁱⁱIn July 2010, the final meaningful use regulations were released. More information is available at http://healthit.hhs.gov.

References

1. Bates DW, Gawande AA. Improving safety with information technology. N Engl J Med 2003;348:2526–34. [PubMed]

2. Field TS, Rochon P, Lee M, et al. Computerized clinical decision support during medication ordering for long-term care residents with renal insufficiency. J Am Med Inform Assoc 2009;16:480–5. [PMC free article] [PubMed]

3. Schedlbauer A, Prasad V, Mulvaney C, et al. What evidence supports the use of computerized alerts and prompts to improve clinicians' prescribing behavior? J Am Med Inform Assoc 2009;16:531–8. [PMC free article] [PubMed]

4. Rind DM, Safran C, Phillips RS, et al. Effect of computer-based alerts on the treatment and outcomes of hospitalized patients. Arch Intern Med 1994;154:1511–17. [PubMed]

5. Osheroff JA, Teich JM, Middleton B, et al. A roadmap for national action on clinical decision support. J Am Med Inform Assoc 2007;14:141–5. [PMC free article] [PubMed]

6. Nebeker JR, Hoffman JM, Weir CR, et al. High rates of adverse drug events in a highly computerized hospital. Arch Intern Med 2005;165:1111–16. [PubMed]

7. US Department of Health and Human Services CDS Government Collaboratory. http://www.healthit.hhs.gov/portal/server.pt?open=512&mode=2&cached=true&objID=1230 (accessed 15 Jan 2010).

8. Agency for Healthcare Research and Quality AHRQ National Resource Center for Health IT. Washington, DC: Agency for Healthcare Research and Quality; http://www.healthit.ahrq.gov/ (accessed 15 Feb 2009).

9. US Department of Health & Human Services Personalized health care. Washington, DC: US Department of Health and Human Services; http://www.hhs.gov/myhealthcare (accessed 1 Jan 2010).

10. National Priorities Partnership National priorities partnership. http://www.nationalprioritiespartnership.org/PriorityDetails.aspx?id=612 (accessed 30 Mar 2009).

11. Osheroff JA, editor. , ed. Improving medication use and outcomes with clinical decision support: a step-by-step guide. Chicago, IL: HIMSS, Scottsdale Institute, AMIA, ISMP, ASHP, AMDIS, 2009.

12. Greenes RA. Clinical decision support: the road ahead. Amsterdam, Boston: Elsevier Academic Press, 2007.

13. Agency for Healthcare Research and Quality Challenges and Barriers to Clinical Decision Support (CDS) design and implementation experienced in the Agency for Healthcare Research and quality CDS demonstrations (Prepared for the AHRQ National Resource Center for Health Information Technology under Contract No. 290-04-0016.). Rockville, MD: Agency for Healthcare Research and Quality, 2010.

14. Wright A, Bates DW, Middleton B, et al. Creating and sharing clinical decision support content with Web 2.0: issues and examples. J Biomed Inform 2009;42:334–46. [PubMed]

15. Agency for Healthcare Research and Quality The AHRQ clinical decision support initiative. Washington, DC: Agency for Healthcare Research and Quality, http://www.healthit.ahrq.gov/portal/server.pt?open=512&objID=654&&PageID=13665&mode=2&in_hi_userid=3882&cached=true (accessed 22 Jan 2010).

16. Shiffman RN, Karras BT, Agrawal A, et al. GEM: a proposal for a more comprehensive guideline document model using XML. J Am Med Inform Assoc 2000;7:488–98. [PMC free article] [PubMed]

17. Shiffman RN, Michel G, Essaihi A, et al. Bridging the guideline implementation gap: a systematic, document-centered approach to guideline implementation. J Am Med Inform Assoc 2004;11:418–26. [PMC free article] [PubMed]

18. Partners Healthcare [Internet] Partners Healthcare. Clinical Decision Support Consortium Website. http://www.partners.org/cird/cdsc/ (accessed 18 Jan 2010).

19. Agency for Healthcare Research and Quality CERTs Centers. Rockville, MD: Agency for Healthcare Research and Quality; http://www.certs.hhs.gov/centers/centers.htm#14 (accessed 1 Mar 2010).

20. American Hospital Association Report on the capital crisis: impact on hospitals. Washington, DC: American Hospital Association, 2009.

21. NCR Corporation Healthcare informatics research series: the economy's impact on healthcare IT spending. Duluth, GA: NCR Corporation, 2009.

22. American Recovery and Reinvestment Act of 2009. Pub. L. 111–5, 2009. February 17.

23. US Department of Health & Human Services The ONC-coordinated federal health IT strategic plan: 2008–2012. Washington, DC: US Department of Health & Human Services, 2008.

24. Regulations.gov Medicare and medicaid programs: electronic health record incentive program. Washington, DC: Regulations.gov, 2010. http://www.regulations.gov/search/Regs/home.html#documentDetail?R=0900006480a7c4a8(accessed 1 Mar 2010).

25. US Department of Health and Human Services [Internet] Health information technology: initial set of standards, implementation specifications, and certification criteria for electronic health record technology. Washington, DC: US Department of Health and HumanServices, 2010. http://www.healthit.hhs.gov/portal/server.pt?open=512&objID=1456&parentname=CommunityPage&parentid=31&mode=2&in_hi_userid=11113&cached=true (accessed 1 Mar 2010).

26. Agency for Healthcare Research and Quality Clinical decision support town hall meeting summary report. Washington, DC: Agency for Healthcare Research and Quality, 2009.

27. Sittig DF, Wright A, Osheroff JA, et al. Grand challenges in clinical decision support. J Biomed Inform 2008;41:387–92. [PMC free article] [PubMed]

28. Computational technology for effective health care: immediate steps and strategic directions. Washington, DC: National Academies Press, 2009.

29. National Institutes of Health [Internet] Recovery act limited competition: NIH challenge grants in health and science research (RC1). Washington, DC: National Institutes of Health, 2010. http://www.grants.nih.gov/grants/guide/rfa-files/RFA-OD-09-003.html (accessed 22 Jan 2010).

30. Agency for Healthcare Research and Quality [Internet] Exploratory and developmental grant to improve health care quality through health information technology (IT) (R21). Washington, DC: Agency for Healthcare Research and Quality, 2010. http://www.grants.nih.gov/grants/guide/pa-files/PAR-08-269.html (accessed 1 Mar 2010).

31. Agency for Healthcare Research and Quality [Internet] Utilizing health information technology (IT) to improve health care quality (R18). Washington, DC: Agency for Healthcare Research and Quality, 2010. http://www.grants.nih.gov/grants/guide/pa-files/PAR-08-270.html (accessed 1 Mar 2010).

32. US Department of Health & Human Services [Internet] Strategic Health IT Advanced Research Projects (SHARP) program. Washington, DC: US Department of Health & Human Services, 2010. http://www.healthit.hhs.gov/portal/server.pt?open=512&objID=1436&mode=2 (accessed 1 Mar 2010).

Articles from Journal of the American Medical Informatics Association : JAMIA are provided here courtesy of
American Medical Informatics Association