A Framework for evaluating the costs, effort, and value of nationwide health information exchange

doi:10.1136/jamia.2009.000570

J Am Med Inform Assoc. 2010 May-Jun; 17(3): 295–301.

doi: 10.1136/jamia.2009.000570

PMCID: PMC2995720

A Framework for evaluating the costs, effort, and value of nationwide health information exchange

Brian E Dixon,¹ Atif Zafar,² and J Marc Overhage²

¹Regenstrief Institute, Inc and Indiana University School of Informatics, Indianapolis, Indiana, USA

²Regenstrief Institute, Inc and Indiana University School of Medicine, Indianapolis, Indiana, USA

Corresponding author.

Correspondence to Mr Brian E Dixon, Regenstrief Institute, Inc, 410 W 10th Street, Suite 2000, Indianapolis, IN 46202, USA; Email: bdixon/at/regenstrief.org

Disclaimer The opinions expressed in this manuscript are those of the authors and do not necessary reflect those of the US Department of Health and Human Services (HHS), the Office of the National Coordinator for Health Information Technology, or any other agent of the US Government.

Received July 7, 2009; Accepted March 3, 2010.

This article has been cited by other articles in PMC.

Abstract

Objective

The nationwide health information network (NHIN) has been proposed to securely link community and state health information exchange (HIE) entities to create a national, interoperable network for sharing healthcare data in the USA. This paper describes a framework for evaluating the costs, effort, and value of nationwide data exchange as the NHIN moves toward a production state. The paper further presents the results of an initial assessment of the framework by those engaged in HIE activities.

Design

Using a literature review and knowledge gained from active NHIN technology and policy development, the authors constructed a framework for evaluating the costs, effort, and value of data exchange between an HIE entity and the NHIN.

Measurement

An online survey was used to assess the perceived usefulness of the metrics in the framework among HIE professionals and researchers.

Results

The framework is organized into five broad categories: implementation; technology; policy; data; and value. Each category enumerates a variety of measures and measure types. Survey respondents generally indicated the framework contained useful measures for current and future use in HIE and NHIN evaluation. Answers varied slightly based on a respondent's participation in active development of NHIN components.

Conclusion

The proposed framework supports efforts to measure the costs, effort, and value associated with nationwide data exchange. Collecting longitudinal data along the NHIN's path to production should help with the development of an evidence base that will drive adoption, create value, and stimulate further investment in nationwide data exchange.

Keywords: Computer communication networks, evaluation studies as topic, medical informatics, United States

Health information exchange (HIE) is the sharing of clinical and administrative healthcare data among healthcare institutions, providers, and data repositories.¹ The broadest use of the term HIE includes one physician office faxing a patient's medical records to another physician office. Using such a definition makes HIE a common activity in modern medicine and nursing as healthcare professionals, institutions, and information systems often have a need to exchange data or information. Less common are organized entities that specialize in facilitating electronic HIE (referred to as HIEs) among a diverse group of often competing healthcare system stakeholders, such as hospitals and physician practices.² This helps distinguish HIEs from other applications of informatics that support HIE in its broad sense. Such entities help bridge the cultural, technical, and financial barriers that may hinder the wider adoption of electronic HIE in a fragmented healthcare delivery system.

Several HIEs have developed successful infrastructures for exchanging electronic data among disparate healthcare institutions within a geographical region or state.^3–7 The success of regional and state initiatives has inspired the US government, along with a number of public and private stakeholders, to develop an interoperable nationwide health information network (NHIN). The NHIN is designed to securely link regional and state HIEs to create a national, interoperable ‘network of networks’ for sharing healthcare data in the USA.⁸ ⁹ Connections between individual HIEs will enable data to be accessible to any provider participating in any HIE when caring for a patient who has data scattered across the fragmented US healthcare system.

In 2007, the Office of the National Coordinator for Health IT (ONC), part of the US Department of Health and Human Services, awarded several contracts to develop the infrastructure necessary for the NHIN.¹⁰ Indiana University and the Regenstrief Institute received one of those contracts. Although the majority of the work performed under these contracts focused on advancing the governance and technical frameworks that enable operational data exchange on the NHIN, participation in the NHIN cooperative allowed us to advance our thinking on the evaluation of health information exchange.

Evaluation of health information exchange is important, because it enables HIE organizations to demonstrate value to their stakeholders. The broader informatics community is also interested in evaluation, because it enables generalization across HIEs, states, and regions. Unfortunately, evaluation reports in published studies as well as general frameworks and methods for evaluating health information exchange are rare.¹¹ Additional evaluation frameworks, methods, and reports may spur additional HIE activity, including investment in and adoption of HIE standards and services.

During the course of our NHIN contract, we sought to evaluate the cost, effort, and value of exchanging data between regional HIEs via the NHIN. Our efforts produced a general framework for evaluating health information exchange involving the NHIN. We subsequently presented our framework to stakeholders in HIEs across the USA to assess it and collect feedback for its improvement. In this paper we describe the framework and the results of the survey. We then discuss how the framework might be used by the informatics community to evaluate local and national HIE activities.

Background

In 2005, ONC established four consortia to architect a standards-based, nationwide network for health data exchange.⁹ The consortia developed a technical framework that defined several core NHIN services which included, but was not limited to, locating and retrieving information, providing consumers with access to personal health records, and identity management. The consortia were followed by 16 contractors and grantees that developed ‘production-ready’ systems which implemented the various core services. The technical framework was also enhanced to include detailed implementation specifications¹² to support expanding the NHIN to include additional HIEs.

In parallel, the NHIN has developed a comprehensive governance framework. The primary artifact of the governance work is the Data Use and Reciprocal Sharing Agreement (DURSA), a legal framework authorizing the exchange of protected health information across the NHIN. The DURSA, at its core, provides privacy and security for the information exchanged.¹³ It further describes the governance of the NHIN and the requirements for those entities which desire to become part of the NHIN.

The technical, legal, and governance frameworks create a pathway for operational data exchange to occur between HIEs in the near future. However, it remains unclear how many HIEs will become early participants in an operational NHIN. There are three key potential barriers to early participation.

First, there are simply very few operational HIEs sharing data within their communities.¹⁴ Despite valiant efforts on the part of community organizers and investment from federal and state governments, many HIEs can currently only exchange trial results.¹⁵ ¹⁶ Some HIEs have shut down or disbanded.¹⁷ Young HIEs are unlikely to have the capacity to develop and implement additional systems and interfaces for national data exchange when their local efforts are just getting off the ground.

Second, it is unclear how many HIEs can afford national data exchange. HIEs are often dependent on grants to support their development,¹⁶ ¹⁷ and it is uncertain what percentage of the American Recovery and Reinvestment Act of 2009 funds will be used to pay for the adoption of NHIN-ready technologies and interfaces. So even if young HIEs desired to join the NHIN, it is questionable whether these organizations would have access to the funds to do so.

Finally, the value proposition for national health information exchange is not yet fully understood. HIE evaluation itself is a nascent sub-discipline within the field of informatics. The United Hospital Fund convened a meeting in 2006 to explore the issues in HIE evaluation.¹¹ The meeting, and subsequent works from meeting attendees, has advanced the measurement of safety, quality, efficiency, and costs of care within the context of HIE. However, it was beyond the scope of the meeting to examine the incremental value of national data sharing. Incremental value would, for example, measure the benefits of sharing data between New York and California versus the benefits of sharing data between hospitals within New York City. The Center for IT Leadership estimated that the potential value for national health information exchange might be as high as US$77.8 billion per year.¹⁸ However, the model used for this estimation contained many assumptions. It further aggregated benefits and did not address the question of the incremental value of sharing data among individual healthcare organizations compared with regions.

Our goal was to develop an evaluation framework that would complement the NHIN technical, legal, and governance frameworks. The framework would help the informatics community address uncertainties involving the costs, effort, and value of the NHIN. We sought to create a general framework that could help the NHIN cooperative and other HIEs collect data that would help chart progress along the pathway to production and beyond. In the following section, we present an evaluation framework for nationwide HIE. Then we describe the results of a survey assessing the framework. Finally, we discuss how the framework can evolve and be used to advance HIE activities and HIE evaluation.

Model formulation

The evaluation framework was developed in an iterative fashion. The authors initially met to discuss an approach to developing a framework that would satisfy the criteria of the NHIN contract awarded by ONC. That discussion ended with a decision to create a general framework that could be used in the future by other NHIN participants as well as other HIEs and informatics researchers. The primary architect of the framework (AZ) developed several iterative drafts of the framework over a 6-month period. The other authors (BED, JMO) reviewed the drafts and provided electronic comments and suggestions to the primary architect.

The framework is based on the previous work in HIE evaluation and the authors' experiences in developing and operating a real-world statewide health information exchange. The authors reviewed earlier work from a number of sources, including the peer-reviewed literature,¹¹ ^19–21 the Markle Foundation,²² the eHealth Initiative,²³ and the AHRQ National Resource Center for Health Information Technology.²⁴ The various models and frameworks developed previously were synthesized and edited to reflect the nature of the interface specifications under development by the NHIN cooperative. The goal was to create a comprehensive framework that would ‘guide in the selection from among the many possible factors that can be studied’¹¹ in the course of developing and using the NHIN. The authors intended the framework to be used formatively to measure costs, effort, and value as the NHIN matures. Furthermore, the framework builds on previous work by incorporating individual HIE factors and providing for unique factors that are relevant only when exchanging data among a group of HIEs.

Model description

The final evaluation framework is summarized in figure 1. It consists of five distinct categories that address various aspects of nationwide data exchange, including: (1) implementation; (2) technology choices; (3) policy issues; (4) issues regarding the data being shared; and (5) the value derived from the shared data. These core evaluation dimensions span the entire scope of the NHIN deployment effort, from the early planning phases to implementation to measuring the derived value from the data exchange activities. Within each broad dimension are more detailed metrics and suggestions that can be used to evaluate the costs, level of effort, and value for a given HIE. The complete evaluation framework is included as appendix A, available as an online only data supplement.

Figure 1

A framework for evaluating nationwide health information exchange. HIE, health information exchange; HIEs, organized entities that specialize in facilitating electronic HIE; NHIN, nationwide health information network.

Implementation

The first section provides a list of metrics and suggestions for evaluating aspects of implementing the NHIN. The technical and governance frameworks of the NHIN call for each HIE organization to implement a defined set of service interfaces. Some of these interfaces are ‘core services’ that enable such functions as patient identity matching, summary data exchange, and audit logging. Other interfaces provide for the exchange of specific data types. For example, Indiana received a specific task order to examine the services and data necessary to share patients' medication histories between HIEs for performing outpatient medication reconciliation. We further examined specific transactions required to report aggregate-level quality and biosurveillance data to federal agencies such as the Centers for Medicare and Medicaid Services and the Centers for Disease Control and Prevention. The metrics chosen for inclusion in the implementation section enable HIEs to measure the cost and effort required to develop and instantiate the various services called for in the technical and governance frameworks.

Technology

The NHIN is a network of networks that is designed to be technology independent. This means that each HIE organization, which serves as a node on the network, is free to implement its own suite of services, applications, and technologies. A comprehensive evaluation framework must account for the various choices that an individual HIE organization makes, because those choices impact performance, costs, and the effort required to implement the NHIN services. The technological metrics assess an individual HIE organization's technology choices and their impact on the NHIN in terms of performance, cost, and effort. For example, patient identity matching requires not only implementing the ability to send and receive patient queries but the backend systems to match patients within the local HIE network and return a response. Those HIEs using a record locater service may have different performance times, costs, and integration efforts from an HIE organization that maintains a central repository of all patient data. The measures in this section were designed to tease out the impact of technology choices on NHIN implementation and operation.

Policy

The third category of measures examines NHIN policies and their relationships to participating HIEs. Although the DURSA and NHIN governance framework enable HIEs to securely exchange protected health information between HIEs, they may impact local policies and procedures. For example, the user authentication scheme used in NHIN messaging may require local HIEs to change their identity management scheme, or it may introduce new user codes and roles that were not utilized previously. Reviewing the DURSA, and helping local HIE stakeholders accept its provisions, may require assistance from legal and policy consultants. The measures chosen for this section were designed to support HIEs in documenting and examining aspects of the governance framework, including the effort necessary to bring HIEs ‘on board’ from a policy perspective.

Data

The fourth category focuses on the quality and quantity of the data elements exchanged among HIEs. The NHIN technical framework outlines the minimum requirements for data exchange, namely the query and retrieval of continuity of care documents (CCD). The minimum requirements for a CCD are outlined in documents available from Health Level 7 and the Health Information Technology Standards Panel. Any required testing to join the NHIN emphasizes only the message structure and data format. Little is provided in the technical framework to ensure high quality data are exchanged across the network. The measures in this category pertain to data quality and the effort required to transform local data, often stored by providers using local data dictionaries, into standardized data mapped to normalized dictionaries (eg, SNOMED CT, LOINC). These aspects of data exchange are important for successful, sustainable HIE. However, they are hard to achieve and are rarely reported in the literature.²⁵ ²⁶

Value

The final category contains measures to assess the value that NHIN data exchange brings to the nation, states, and the participating communities. This category contains many of the more complex metrics associated with informatics evaluation, including patient outcomes, safety, and return on investment. It further provides for qualitative measurements of satisfaction, including patients and providers. Finally, the category contains a placeholder for metrics that describe the incremental value in the exchange of data beyond the local community.

Model assessment

To assess the usefulness of the evaluation framework, the authors created an online survey. Because we intended the framework to be useful to both current and future NHIN participants, we targeted the broad HIE community, which included other NHIN contractors as well as HIEs that were not receiving funds from ONC to develop parts of the NHIN. The survey was designed to obtain feedback from HIE leaders across the USA regarding their perceived usefulness of the metrics specified in the framework. The final version of the survey instrument is included as appendix B, available as an online only data supplement.

It is important to note that our intention was not to place a value on any one metric or set of metrics by asking respondents to evaluate the framework. Various stakeholders in an HIE organization are concerned with different sets of outcomes. Some stakeholders may prefer efficiency metrics over patient safety metrics, whereas other stakeholders may prefer financial metrics over utilization metrics. The survey was designed to sample HIE leaders to determine if our perceptions of valuable metrics aligned with theirs. Individual providers, payers, and other stakeholders within a HIE have different reasons for participating. This often influences the choice of metrics selected by HIE leadership when demonstrating value to their mix of stakeholders. The framework used in the survey was designed to be broad so as to represent as many metric categories as thought to be useful when evaluating the NHIN.

The survey was conducted online using SurveyMonkey (Menlo Park, California, USA). The authors identified a non-random sample of 32 HIEs composed of participants in the NHIN cooperative and other HIEs identified through previous efforts of one author (BED) as a part of the AHRQ National Resource Center for Health IT.²⁷ The HIEs targeted were drawn from a diverse pool of HIEs at various stages in the planning or implementation process. The authors subsequently identified appropriate individuals from these HIEs to complete the survey. The individuals identified were perceived by the authors to have an appropriate level of understanding of the scope and vision of their respective HIE.

The survey research project was approved by the Indiana University–Purdue University Indianapolis (IUPUI) and Clarian Institutional Review Board. E-mail invitations to participate in the survey were sent in June of 2008. Up to three reminder e-mails were sent in July and August of 2008. Phone calls were also made to remind individuals about the survey.

The metrics assessed in the survey were divided into their corresponding framework categories. Respondents rated each metric within each category using a Likert scale, in which a value of one represented very useful and a value of four represented not useful. Respondents further ranked their top three metrics in each category. However, the ranking scores were eventually dropped from the analysis as the data were determined to be generally unhelpful in assessing the framework.

Twenty-six of the 32 (81%) surveys were ultimately completed. Three of the surveys contained incomplete data and were dropped from the analysis. Seven (30%) of the surveys were completed by HIEs self-described as non-operational. Eight (35%) of the respondents represented HIEs in operation for less than 2 years. Finally, eight (35%) of the respondents represented HIEs in operation for at least 2 years.

Data on the individual metrics are summarized in table 1. Respondents' mean usefulness scores were fairly consistent, ranging from 1.3 to 2.7 with a SD of 0.7. Therefore, nearly all of the metrics and categories were individually rated as useful or very useful. The only category that stood out from the others in terms of usefulness was the value category. The mean scores for the individual metrics and metric classes within the value group were generally viewed as very useful. This was not surprising because the primary goal of evaluation is to demonstrate value.

Table 1

Mean usefulness scores, in descending order of usefulness

In addition to summarizing individual metric scores, the authors performed a subgroup analysis. Respondents were divided into two groups¹: those actively developing NHIN prototypes and components and² all other HIEs that completed the survey. The subgroup analysis was performed to detect differences in opinion between those working on the NHIN and potential future members of the NHIN. As expected, there were differences between the two groups.

The far right column in table 1 indicates where the authors felt there were significant differences between the two groups. A 0.5 mean score differential was selected by the authors as the determinant for significance. Such a difference was chosen because it represents half the distance between, for example, useful and somewhat useful.

One area of difference between the groups was observed for cost-related measures. Avoided costs, financial indicators, and NHIN service implementation costs were all judged more useful to non-NHIN HIEs than those participating in the NHIN. Similarly, those measures directly associated with the NHIN were judged more useful by non-NHIN HIEs. NHIN cost and effort measures appear to be less useful to those HIEs actively engaged in the NHIN. A likely cause for these differences is the fact that the NHIN participants surveyed were being reimbursed for their time and effort in transitioning from an independent HIE network to a NHIN cooperative HIE member (NHIE). The leadership at an NHIE may be less concerned with NHIN transition costs because investment in the NHIN was not coming out of the HIE's bottom line. Equally as likely is a concern on the part of those completing the survey from non-NHIN HIEs that future efforts to join the NHIN would require them to ‘sell the idea’ to their boards or management committees. This may have prompted them to rank financial indicators as more useful because they have little information on which to base return on investment estimates. Furthermore, measuring the impact of NHIN architectural choices may not be important to current NHIN participants because they have already spent the time and effort to enhance their HIEs to accommodate the NHIN and its technical framework. These measures are perceived, however, by non-NHIN HIEs to be very useful when planning for future NHIN adoption.

In addition to measuring the usefulness of metrics included in the evaluation framework, we asked respondents to tell us what evaluation measures they are currently capturing as well as those they intend to collect in the future. We summarize these responses in table 2.

Table 2

Current and future measures written in by respondents.

The following measure classes had the highest reported usage currently by the responding HIEs: utilization (eg, number of data exchange transactions); participation (eg, engagement of community stakeholders); clinical outcomes (eg, quality indicators); financial (eg, cost avoidance); and users (eg, number of providers part of the HIE). Clinical process measures, including workflow, and measures of provider or patient satisfaction had the lowest reported current usage. Clinical outcomes, financial measures, measures of satisfaction, and clinical process measures had the highest reported future usage.

These results follow an expected pattern in health information exchange. Younger HIEs, especially those that are not yet sharing data, tend to measure success in terms of numbers of stakeholders at the table or the number of providers that are involved in exchange activities. As an HIE organization matures, it is able to measure more complicated aspects, including clinical outcomes, clinical process outcomes, financial savings, and user satisfaction. This is reflected in the information provided by the survey respondents. Less than half of the respondents indicated that their HIEs were currently using financial measures to evaluate activities, yet more than half indicated that their HIEs planned to use financial measures in the future. Clinical outcomes measures followed a similar trend. On the other hand, more than half of respondents indicated that their HIEs utilize participation measures, but few respondents indicated that they plan to use such measures in the future.

Discussion

Evaluation is critical to the success of health information exchange, because it provides evidence on the costs, effort required, and value of data exchange. Evaluation efforts and reported data on HIE to date have been sparse.¹¹ To support HIE evaluation efforts, the authors present an evaluation framework for the NHIN. The framework helps HIEs consider available measures and measure categories for assessing the costs, effort, and value of nationwide health information exchange. An initial assessment of the framework with a group of HIEs across the USA reveals that the included measures and categories are generally perceived to be useful.

The framework was designed to be a formative evaluation tool. In formative evaluation of HIE and other community-based initiatives, measuring a base hit is just as important as capturing evidence on a home run.²⁸ Formative evaluation has been suggested by a number of informatics researchers as a valid method by which to incrementally capture and report data crucial to demonstrate the value of a technology such as electronic health records and health information exchange.^19–21 The evaluation framework described here contains a range of useful measures for progressively capturing evidence during each stage of an HIE's development lifecycle, making it useful as a formative tool. Furthermore, the framework possesses a mixture of quantitative and qualitative measures, both of which are required in formative evaluation.²⁰

The framework further builds on the work of community-based HIE evaluation efforts. Data exchange, no matter the scope, involves similar functions, including patient matching, privacy protections, and the need for standardized vocabularies. Therefore, this framework was not developed de novo, but it extends traditional models used in local exchange efforts ⁴ ⁷ by incorporating local exchange measures. In addition to previously described HIE measures, the framework defines NHIN-specific technology choices, implementation efforts, and policy considerations. These additional components make the framework NHIN-centric, distinguishing it from previous frameworks designed for generic HIE evaluation. These components further allow HIEs to evaluate distinct activities enabled by the NHIN technical and policy frameworks, enabling the data captured to be reported separately from those pertaining to local HIE development, operations, and sustainability.

The framework further extends existing HIE evaluation work by incorporating the notion of incremental value. Incremental value metrics are crucial to the NHIN, because the premise of nationwide exchange rests on the assumption that sharing data across regions is valuable. Otherwise, HIEs should simply concentrate on exchanging data within their communities because ‘all healthcare is local’.²⁹ In Indiana, we have started to measure the incremental value of nationwide HIE for patients who seek care at Indiana-based facilities. Our statewide syndromic surveillance system, which captures real-time data feeds from 76 emergency departments throughout Indiana, is one source of data for this analysis. Preliminary data reveal that nearly 5% of emergency department encounters are for patients who live outside Indiana. These patients do not have data in our local HIE, the Indiana Network for Patient Care, making them primary candidates for data requests using the NHIN. Similarly, we are examining migratory patterns of patients, as new patients to the area might also be primary beneficiaries of an NHIN. Efforts such as these are necessary to quantify and qualify statements that nationwide HIE will benefit not only patients as they seek care from a complex healthcare environment but also the clinicians who desire to provide high quality care to their patients.

Ultimately, any evaluation framework will at best provide only guidance.³⁰ The framework described here was never intended to be the final guide for HIE and NHIN evaluation efforts. Whereas survey respondents felt the measures in the framework are useful, there are other measures that are equally as useful. For example, additional measures for workflow and clinical process would most likely enhance the framework and address the often forgotten issues surrounding informatics applications integrating into clinical workflow.³¹ We view the framework as a starting point, and we hope the informatics community will work with us to refine and enhance the framework as HIE activities and the NHIN mature over time. Such work will come with iterations in NHIN development and additional work by HIEs across the USA to evaluate their activities and report data on the costs and benefits of HIE.

Conclusion

The NHIN is currently on a path toward production. Several pilot projects are to be initiated or completed in 2010 that will lay the groundwork for continued development and expansion of the NHIN.³² ³³ Technical, legal, and governance frameworks make it feasible for real-world data exchange via an NHIN in the near future. These frameworks may further make it possible for HIEs to quickly and efficiently support providers as they seek to achieve ‘meaningful use’ as defined in draft regulations from the Centers for Medicare and Medicaid Services³⁴ and qualify for incentive payments outlined in the American Recovery and Reinvestment Act of 2009.³⁵ Despite notions that the NHIN should be a health internet that simply facilitates point-to-point communication between healthcare industry stakeholders,³⁶ more value may be provided by a NHIN that leverages local and regional HIE networks to scale complexity, costs, and outcomes. As the NHIN is further developed, implemented, and adopted, it should continue to be studied and evaluated. A formative approach using the proposed evaluation framework may enable HIEs and the NHIN to demonstrate value along the path. This will help develop a strong evidence base that will drive adoption and further investment in the NHIN and related HIE activities. Without study and evaluation it will be impossible to tell which models, services, and standards best support nationwide HIE, and it will be difficult to assign value in terms of quality, safety, or costs to the NHIN.

Supplementary Material

Web Only Data

Click here to view.

Acknowledgments

The authors would like to thank Yan Ding and Xiaochun Li for their statistical and analytical support with the survey data. They further thank Shaun Grannis and Theda Miller for their comments on early versions of the framework and manuscript. The authors also thank the anonymous reviewers who provided encouragement and guidance on revising the manuscript before publication.

Footnotes

Funding: Portions of this work were funded under US Department of Health and Human Services Contract HHSP23320074102EC.

Competing interests: None.

Ethics approval: The survey research project was approved by the Indiana University–Purdue University Indianapolis (IUPUI) and Clarian Institutional Review Board.

Provenance and peer review: Not commissioned; externally peer reviewed.

References

1. AHRQ Health Information Exchange. 2009. http://www.healthit.ahrq.gov/hie (accessed Jan 2010).

2. Dixon BE, Scamurra S. Is there such a thing as healthy competition? Annual HIMSS Conference and Exhibition; 2007. New Orleans, LA: HIMSS, 2007.

3. Biondich PG, Grannis SJ. The Indiana network for patient care: an integrated clinical information system informed by over thirty years of experience. J Public Health Manag Pract 2004;10:S81–6. [PubMed]

4. Rudin RS, Simon SR, Volk LA, et al. Understanding the decisions and values of stakeholders in health information exchanges: experiences from Massachusetts. Am J Public Health 2009;99:950–5. [PubMed]

5. Goroll AH, Simon SR, Tripathi M, et al. Community-wide implementation of health information technology: the Massachusetts eHealth Collaborative experience. J Am Med Inform Assoc 2009;16:132–9. [PMC free article] [PubMed]

6. Frisse ME, King JK, Rice WB, et al. A regional health information exchange: architecture and implementation. AMIA Annu Symp Proc 2008:212–16. [PMC free article] [PubMed]

7. Johnson KB, Gadd CS, Aronsky D, et al. The MidSouth eHealth Alliance: use and impact in the first year. AMIA Annu Symp Proc 2008:333–7. [PMC free article] [PubMed]

8. Halamka J, Overhage JM, Ricciardi L, et al. Exchanging health information: local distribution, national coordination. Health affairs (Project Hope) 2005;24:1170–9. [PubMed]

9. Rishel W, Riehl V, Blanton C. Summary of the NHIN Prototype Architecture Contracts. Washington, DC: US Department of Health and Human Services, 2007.

10. DHHS Nationwide Health Information Network (NHIN): trial implementations. http://www.hhs.gov/healthit/healthnetwork/trial/ (accessed 2 Feb 2009).

11. Hripcsak G, Kaushal R, Johnson KB, et al. The United Hospital Fund meeting on evaluating health information exchange. J Biomed Inform 2007;40(Suppl 6):S3–10. [PMC free article] [PubMed]

12. DHHS Nationwide Health Information Network (NHIN): resources. 2009. http://healthit.hhs.gov/portal/server.pt?open=512&objID=1194&parentname=CommunityPage&parentid=1&mode=2&in_hi_userid=10741&cached=true (accessed June 9, 2009).

13. Gravely SD, Whaley ES. The next step in health data exchanges: trust and privacy in exchange networks. J Healthc Inf Manag 2009;23:33–7. [PubMed]

14. Overhage JM, Evans L, Marchibroda J. Communities' readiness for health information exchange: the National Landscape in 2004. J Am Med Inform Assoc 2005;12:107–12. [PMC free article] [PubMed]

15. Adler-Milstein J, Bates DW, Jha AK. U.S. Regional health information organizations: progress and challenges. Health Aff (Millwood) 2009;28:483–92. [PubMed]

16. Adler-Milstein J, McAfee AP, Bates DW, et al. The state of regional health information organizations: current activities and financing. Health Aff (Millwood) 2008;27:w60–9. [PubMed]

17. Miller RH, Miller BS. The Santa Barbara County care data exchange: what happened? Health Affairs (Project Hope) 2007;26:w568–80. [PubMed]

18. Walker J, Pan E, Johnston D, et al. The value of health care information exchange and interoperability. Health Aff (Millwood) Suppl Web Exclusives. 2005;24:W5-10–18. [PubMed]

19. Johnson KB, Gadd C. Playing smallball: approaches to evaluating pilot health information exchange systems. J Biomed Inform 2007;40(Suppl 6):S21–6. [PubMed]

20. Ash JS, Guappone KP. Qualitative evaluation of health information exchange efforts. J Biomed Inform 2007;40(Suppl 6):S33–9. [PMC free article] [PubMed]

21. McGowan JJ, Cusack CM, Poon EG. Formative evaluation: a critical component in EHR implementation. J Am Med Inform Assoc 2008;15:297–301. [PMC free article] [PubMed]

22. Markle Foundation The common framework. 2006. http://www.connectingforhealth.org/commonframework/index.html (accessed 10 Jun 2009).

23. eHealth Initiative HIE value and sustainability model. 2008. http://ehealthinitiative.org/toolkit/VSM.mspx (accessed 10 Jun 2009).

24. Cusack CM, Poon EG. Health information technology evaluation toolkit. Rockville, MD: Agency for Healthcare Research and Quality, 2007. Report No: 08-0005-EF.

25. Zafar A, Dixon BE. Pulling back the covers: technical lessons of a real-world health information exchange. Stud Health Technol Inform 2007;129:488–92. [PubMed]

26. Vreeman DJ, Stark M, Tomashefski GL, et al. Embracing change in a health information exchange. AMIA Annu Symp Proc 2008:768–72. [PMC free article] [PubMed]

27. Dixon BE, Welebob EM, Dullabh P, et al. Summary of the Status of Regional Health Information Exchanges (RHIOs) in the United States. 2007. http://healthit.ahrq.gov/portal/server.pt/gateway/PTARGS_0_3882_813237_0_0_18/Summary%20of%20the%20Status%20of%20Regional%20Health%20Information%20Exchanges.pdf (accessed May 2009).

28. Friedman CP. “Smallball” evaluation: a prescription for studying community-based information interventions. J Med Libr Assoc 2005;93(Suppl 4):S43–8. [PMC free article] [PubMed]

29. Grossman JM. Health plan competition in local markets. Health Serv Res 2000;35:17–35. [PMC free article] [PubMed]

30. Heathfield HA, Peel V, Hudson P, et al. Evaluating large scale health information systems: from practice towards theory. Proc AMIA Annu Fall Symp 1997:116–20. [PMC free article] [PubMed]

31. Karsh BT. Clinical practice improvement and redesign: how change in workflow can be supported by clinical decision support. Rockville, MD: Agency for Healthcare Research and Quality, 2009.

32. SSA Social security awards nearly $20 million in recovery act contracts for electronic medical records. Report No. 09-0054-EF. 2010. http://www.ssa.gov/pressoffice/pr/nhin0210-pr.htm (accessed Feb 2010).

33. DHHS Nationwide Health Information Network (NHIN): architecture overview 2010. http://healthit.hhs.gov/portal/server.pt/gateway/PTARGS_0_11673_910517_0_0_18/NHIN_Architecture_Overview_Document_v.1.0.pdf (accessed 16 Feb 2010).

34. Centers for Medicare and Medicaid Services Medicare and medicaid programs; electronic health record incentive program; proposed rule. The Federal Register. Washington DC, USA: Office of the Federal Register, National Archives and Records Administration, 2010;75:1843–92.

35. US Public Law The American Recovery and Reinvestment Act of 2009. Pub. L. No. 111-5, 123 Stat. 115 (17 Feb 2009).

36. Rishel W. Simple interop: the health internet node. 2009. http://blogs.gartner.com/wes_rishel/2009/12/15/simple-interop-the-health-internet-node/ (accessed 16 Feb 2010).

Articles from Journal of the American Medical Informatics Association : JAMIA are provided here courtesy of
American Medical Informatics Association