Many national studies that examine CPOE use in hospitals have relied on either HIMSS or Leapfrog data. These datasets have been used to estimate hospital CPOE adoption rates (Ford et al. 2008
), examine hospital characteristics associated with adoption (Teufel, Kazley, and Basco 2009
), and correlate CPOE adoption with medication-related quality outcomes (Jha et al. 2008
; Yu et al. 2009b
; Kazely and Diana 2011). No previous study has empirically examined the HIMSS or Leapfrog datasets, despite the fact that findings from previous studies can influence managerial and policy decisions. In this study, we sought to examine the internal consistency, level of agreement, and responsiveness of the CPOE variables from these datasets.
Overall, respondents to both the Leapfrog and HIMSS datasets have similar organizational characteristics and in many cases overlap. However, our findings indicate minimal agreement between the two datasets regarding which hospitals have adopted CPOE, but adequate consistency within a given dataset from year to year. Both surveys tend to inaccurately estimate overall changes in true hospital CPOE status over time (i.e., responsiveness), but in different ways. Compared with the Leapfrog data, the HIMSS data tend to overestimate increases in adoption over time. Likewise, relative to the HIMSS data, the Leapfrog data have more downward trending estimates for year over year increases in CPOE status. Our findings suggest serious limitations on the use of either dataset for many health services research purposes. However, given the properties of each survey instrument, it would appear that each data source has differing strengths.
statistic compares the expected level of agreement that would be present by chance to the level of agreement present in the data. A κ
of 0 indicates agreement levels expected purely by chance, and a κ
of 1 indicates perfect agreement. The κ
statistics for all years is high enough to reject the hypothesis that the level of agreement between these two surveys is occurring only by chance. However, the κ
statistics are consistently low enough to indicate there is only marginal agreement between the two surveys. Rosner (2006)
suggests that a κ
<0.4 indicates marginal reproducibility, and the κ
statistics indentified in our study range from 0.1 to 0.3. Further, based on our sensitivity analysis, it appears that the level of agreement may not be improving over time. McNemar's test was used to test the probability that the two surveys were equivalent in the proportions of hospitals identified as having CPOE. In each individual year and in the restricted sample, McNemar's test leads us to reject that hypothesis and conclude that the two surveys are indentifying different proportions of hospitals that have CPOE.
The HIMSS dataset shows consistently higher estimates of hospitals with CPOE than does the Leapfrog dataset, while the two datasets are much closer in estimates of those without CPOE. Our analysis is not able to identify the cause of this difference, but it does suggest that either HIMSS data overstate or Leapfrog data understate the true number of hospitals with CPOE. The Leapfrog criteria for reporting fully implemented CPOE is more restrictive than the HIMSS criteria, which is one likely source of the disagreement. Without a consensus definition of what constitutes full CPOE implementation, it may be impossible to determine which is more accurate. However, it seems reasonable to suggest that researchers wishing to avoid the possibility of having high false positives (regarding CPOE status) may be better served utilizing the Leapfrog data. Conversely, if a given researcher's concern is potential false negatives, the HIMSS data may be more appropriate to use in that instance. These qualities of the two surveys offer the researcher the opportunity to conduct sensitivity analyses using both datasets if such an approach is relevant to their study.
Once a hospital reported full CPOE status, the Leapfrog data and the HIMSS data (less so) were able to reidentify the same hospital as reporting full CPOE status in a subsequent year with adequate consistency. The Leapfrog consistency rate (81–86 percent) and the HIMSS consistency rate (62–78 percent) are either within, or approaching, the generally expectable rate of 80–90 percent (Hennekens and Buring 1987
). Because CPOE adoption is a large undertaking requiring significant financial and human resources to implement, one would expect that sudden downgrades from the status of full CPOE would not take place regularly. Nevertheless, a number of hospitals from each dataset reported less than full CPOE status in an immediately subsequent year. This may be the result of legitimate cases where an implementation failure occurs and full CPOE status regresses (see, e.g., the experience of Cedar Sinai Medical Center; Connolly 2005
); or it may reflect variability in how the question is perceived from year to year by a given individual or his/her successor who fills out the survey.
The differences noted in responses by the same organization to an almost identical question (full CPOE status) on the HIMSS and Leapfrog surveys may have something to do with how, and why, these data are collected. First, as noted above, the mission of the Leapfrog and HIMSS organizations are very different. Thus, certain responses on the Leapfrog survey may result in benefits to the organization (e.g., being able to participate in providing care for Leapfrog employer members' health plan beneficiaries) while no such direct incentives exist for specific responses on the HIMSS survey. This difference may result in less exaggeration of CPOE status on the Leapfrog dataset because organizations would be subject to the loss of benefits, sanctions, or public embarrassment in the event of a Leapfrog audit. Similarly, because Leapfrog offers hospitals partial “credit” for different stages of CPOE implementation, hospitals may be further dissuaded from overestimating their current CPOE status, and they may find benefit from more slowly reporting progress. In contrast, fewer forces may influence the accuracy of responses on the HIMSS survey.
Second, differences in responses by the same hospital to the Leapfrog and HIMSS surveys may be a function of market forces. In regions where competition among Leapfrog-participating hospitals is relatively high, responding hospitals may be more prone to err on the side of “meeting the CPOE standard” and thus garner access to profitable patient populations. Hospitals in less competitive markets may not have such pressures to assure their financial well-being. Lastly, while both surveys are voluntary, it is not known who ultimately in the organization provides the response to either survey or whether this person changes from one year to the next.
Our analysis has several limitations worth noting. First, we only utilized 3 years of HIMSS and Leapfrog data. Most results either showed obvious yearly trends (in one direction or the other) or were relatively consistent from year to year. Nevertheless, we recognize that using only 3 years worth of data may be a limiting factor. Second, we examined only variables related to CPOE for each of the datasets. Thus, our findings may not be applicable to other variables collected in either the HIMSS or Leapfrog data. Future studies should address the validity of other variables in these datasets. Third, some aspects of our analysis required that we compare only those hospitals that participated in both surveys in a given year. The loss of some hospitals that report to one survey but not the other could biases our findings. Lastly, we carefully calibrated the questions regarding CPOE status from both surveys so that we examined “full CPOE” in both datasets. Despite our best efforts, we recognize that the two surveys ultimately word their questions differently, which may explain some of the differences in responses we identified.
We recommend that future research in this area examine two key issues. First, we need a clear definition of what constitutes a CPOE system. It may seem obvious, but without a consensus on this question, validating the presence of CPOE will remain highly problematic. Second, a validation of any measure of CPOE—or other health information technologies such as electronic medical records—will require comparison of survey responses with the direct assessment of the presence of CPOE through standard survey validation approaches. Without a gold standard of CPOE use, it is not possible to fully validate the data.
In conclusion, the disagreement between the HIMSS and Leapfrog datasets regarding hospital CPOE status creates a challenge for researchers, practitioners, and policy makers who wish to understand CPOE from a national perspective. Future research is encouraged to overcome limitations in each data source. Currently, notwithstanding our analyses, the HIMSS and Leapfrog surveys remain the most widely available secondary sources of information on CPOE. Importantly, the AHA has recently begun to collect data on CPOE among hospitals in their newly developed Hospital EHR Adoption Database, a supplemental to the AHA Annual Survey (Jha et al. 2009
). While this dataset promises to provide additional valuable information about CPOE (and other health information technology), to our knowledge, it has not undergone a similar assessment as described herein. Both the Leapfrog and HIMSS datasets, as well as the new AHA data, will continue to be among the scant choices for health services researchers interested in this topic for the foreseeable future. In the absence of a true gold standard measure for CPOE, it is our hope that all organizations collecting data used by health services researchers consider strategies to validate their data collection efforts.