To assess the usefulness of the evaluation framework, the authors created an online survey. Because we intended the framework to be useful to both current and future NHIN participants, we targeted the broad HIE community, which included other NHIN contractors as well as HIEs that were not receiving funds from ONC to develop parts of the NHIN. The survey was designed to obtain feedback from HIE leaders across the USA regarding their perceived usefulness of the metrics specified in the framework. The final version of the survey instrument is included as appendix B, available as an online only data supplement.
It is important to note that our intention was not to place a value on any one metric or set of metrics by asking respondents to evaluate the framework. Various stakeholders in an HIE organization are concerned with different sets of outcomes. Some stakeholders may prefer efficiency metrics over patient safety metrics, whereas other stakeholders may prefer financial metrics over utilization metrics. The survey was designed to sample HIE leaders to determine if our perceptions of valuable metrics aligned with theirs. Individual providers, payers, and other stakeholders within a HIE have different reasons for participating. This often influences the choice of metrics selected by HIE leadership when demonstrating value to their mix of stakeholders. The framework used in the survey was designed to be broad so as to represent as many metric categories as thought to be useful when evaluating the NHIN.
The survey was conducted online using SurveyMonkey (Menlo Park, California, USA). The authors identified a non-random sample of 32 HIEs composed of participants in the NHIN cooperative and other HIEs identified through previous efforts of one author (BED) as a part of the AHRQ National Resource Center for Health IT.27
The HIEs targeted were drawn from a diverse pool of HIEs at various stages in the planning or implementation process. The authors subsequently identified appropriate individuals from these HIEs to complete the survey. The individuals identified were perceived by the authors to have an appropriate level of understanding of the scope and vision of their respective HIE.
The survey research project was approved by the Indiana University–Purdue University Indianapolis (IUPUI) and Clarian Institutional Review Board. E-mail invitations to participate in the survey were sent in June of 2008. Up to three reminder e-mails were sent in July and August of 2008. Phone calls were also made to remind individuals about the survey.
The metrics assessed in the survey were divided into their corresponding framework categories. Respondents rated each metric within each category using a Likert scale, in which a value of one represented very useful and a value of four represented not useful. Respondents further ranked their top three metrics in each category. However, the ranking scores were eventually dropped from the analysis as the data were determined to be generally unhelpful in assessing the framework.
Twenty-six of the 32 (81%) surveys were ultimately completed. Three of the surveys contained incomplete data and were dropped from the analysis. Seven (30%) of the surveys were completed by HIEs self-described as non-operational. Eight (35%) of the respondents represented HIEs in operation for less than 2 years. Finally, eight (35%) of the respondents represented HIEs in operation for at least 2 years.
Mean usefulness scores, in descending order of usefulness
In addition to summarizing individual metric scores, the authors performed a subgroup analysis. Respondents were divided into two groups1
: those actively developing NHIN prototypes and components and2
all other HIEs that completed the survey. The subgroup analysis was performed to detect differences in opinion between those working on the NHIN and potential future members of the NHIN. As expected, there were differences between the two groups.
The far right column in indicates where the authors felt there were significant differences between the two groups. A 0.5 mean score differential was selected by the authors as the determinant for significance. Such a difference was chosen because it represents half the distance between, for example, useful and somewhat useful.
One area of difference between the groups was observed for cost-related measures. Avoided costs, financial indicators, and NHIN service implementation costs were all judged more useful to non-NHIN HIEs than those participating in the NHIN. Similarly, those measures directly associated with the NHIN were judged more useful by non-NHIN HIEs. NHIN cost and effort measures appear to be less useful to those HIEs actively engaged in the NHIN. A likely cause for these differences is the fact that the NHIN participants surveyed were being reimbursed for their time and effort in transitioning from an independent HIE network to a NHIN cooperative HIE member (NHIE). The leadership at an NHIE may be less concerned with NHIN transition costs because investment in the NHIN was not coming out of the HIE's bottom line. Equally as likely is a concern on the part of those completing the survey from non-NHIN HIEs that future efforts to join the NHIN would require them to ‘sell the idea’ to their boards or management committees. This may have prompted them to rank financial indicators as more useful because they have little information on which to base return on investment estimates. Furthermore, measuring the impact of NHIN architectural choices may not be important to current NHIN participants because they have already spent the time and effort to enhance their HIEs to accommodate the NHIN and its technical framework. These measures are perceived, however, by non-NHIN HIEs to be very useful when planning for future NHIN adoption.
In addition to measuring the usefulness of metrics included in the evaluation framework, we asked respondents to tell us what evaluation measures they are currently capturing as well as those they intend to collect in the future. We summarize these responses in .
Current and future measures written in by respondents.
The following measure classes had the highest reported usage currently by the responding HIEs: utilization (eg, number of data exchange transactions); participation (eg, engagement of community stakeholders); clinical outcomes (eg, quality indicators); financial (eg, cost avoidance); and users (eg, number of providers part of the HIE). Clinical process measures, including workflow, and measures of provider or patient satisfaction had the lowest reported current usage. Clinical outcomes, financial measures, measures of satisfaction, and clinical process measures had the highest reported future usage.
These results follow an expected pattern in health information exchange. Younger HIEs, especially those that are not yet sharing data, tend to measure success in terms of numbers of stakeholders at the table or the number of providers that are involved in exchange activities. As an HIE organization matures, it is able to measure more complicated aspects, including clinical outcomes, clinical process outcomes, financial savings, and user satisfaction. This is reflected in the information provided by the survey respondents. Less than half of the respondents indicated that their HIEs were currently using financial measures to evaluate activities, yet more than half indicated that their HIEs planned to use financial measures in the future. Clinical outcomes measures followed a similar trend. On the other hand, more than half of respondents indicated that their HIEs utilize participation measures, but few respondents indicated that they plan to use such measures in the future.