This case study provides rich documentation of the outcomes from a successful electronic referral program, and it suggests a range of design features and implementation factors that accounted for the program’s success. Users perceived that eReferral largely prevented the occurrence of low-value specialty visits due to unclear consult questions, incomplete workups, and referrals for problems that could be managed in primary care. The system was also perceived as having markedly reduced wait times for specialty services, which had previously been up to a year for some specialties in this historically under-resourced setting.
These successes were achieved because PCPs and specialty reviewers were willing to spend time using the system rather than circumventing it. For most reviewers, this effort was part of their compensated clinical time. However, PCPs were not compensated for the additional time required to submit and manage referrals, order preliminary tests, and manage or co-manage patients. Some of these tasks were particularly burdensome in clinics with poor technical infrastructure. Likewise, increased workload or time has been reported in other studies of HIT.23, 24
Nonetheless, PCPs (and other users) were generally enthusiastic about eReferral, chiefly due to the professional satisfaction of gaining improved access to specialty care for their patients. PCPs also valued the unique opportunities to learn and to gain reassurance from the dialogue with specialists that eReferral enabled. This result was a direct product of system features that enabled asynchronous communication between the parties to the referral.
Although the primary factor motivating the use of eReferral was its perceived benefits, several other factors played a role in the system’s success, including an intuitive user interface that minimized training needs, the development team’s readiness to add or modify features in response to users’ needs, a measured pace of rollout, strong physician champions, and the distinctive process adaptations used in some clinics (such as use of follow-up tracking sheets). The mandatory nature of the system also undoubtedly promoted adoption. However, even in large-scale technology adoption efforts, users will find ways to get around the system if it is difficult to use or has negative consequences.25
This was rarely observed for eReferral.
The system appeared to have some unintended workflow consequences. One was a shift in administrative tasks to PCPs. While the transfer of tasks such as test-ordering and patient management from specialist to PCP is arguably appropriate, the transfer of administrative tasks to PCPs is inefficient. A second problem was the inability of patients to participate in setting appointment times for their specialty care visits, a particular problem for the large percentage of homeless and limited English-speaking patients in the clinics’ populations. As in other studies,25
users responded to these effects of the system on workflow by adopting supplementary processes and workarounds. These workflow issues were addressed in eReferral through system design changes. Several studies have emphasized the need for analysis of workflow and redesign prior to implementing HIT.26–28
These results also point to the need to continue seeking and adapting to unintended consequences post-implementation.
Overall, the success of eReferral may be most attributable to the “human-centered” approach29
taken in its development. The design team considered workflow at the system development stage. They used participatory design strategies, with clinicians who understood user needs serving as key members of the development and implementation teams. The in-house software team, in combination with ongoing evaluation, enabled continual improvement through system design changes rather than simply through training and workflow adaptations. Coupled with a measured pace of roll-out and strong social influence, the design process may have enabled a virtuous cycle in which each specialty clinic added to eReferral led to design changes that improved usability, thereby enhancing user perceptions and increasing the demand for more specialty clinics to be brought online. This example stands in contrast to current EHR implementation plans in many healthcare organizations, where new systems are being fielded rapidly with limited opportunities to make design changes. Our findings are also consistent with result from a systematic review that found better outcomes from “home-grown” e-prescribing systems.30
An ongoing challenge in implementation of eReferral is ensuring that users become familiar with system changes. Consistent with other studies,31, 32
we found that at least some users were unaware of system features, particularly for changes that had been made to correct initial problems. This occurred despite announcements and demonstrations of these features in outreach meetings, newsletters, and email broadcasts. Bundling changes and rolling them out on a less frequent or a more predictable schedule may be one strategy for helping users stay up-to-date. Another, more focused strategy would be to use system logs to identify users who are not taking advantage of new features in order to target training toward those individuals.
An important limitation of the study is its uncertain generalizability beyond the safety net delivery system in which specialists are salaried and PCPs face sometimes daunting challenges in obtaining specialty access. However, integrated delivery systems such as staff-model health maintenance organizations and possibly newer arrangements such as accountable care organizations have the incentives to compensate specialists and PCPs for the shifts in effort that eReferral would require. In addition, improved communication and coordination with specialists would likely be welcomed enthusiastically by PCPs in most communities. The eReferral model may also be a useful tool in the Patient Centered Medical Home, provided that the incremental PCP time is adequately covered by the coordination fees included in that model. Our study was also limited to interviews and did not employ other usability methods, such as user surveys and observations.29
However, separate surveys of PCPs7
found consistent results. Another possible limitation was that if medical directors selected users who were favorably disposed toward eReferral to participate in the interviews, we may have missed reports of downsides of the system or implementation challenges.
In fee-for-service settings, an eReferral model might be acceptable to specialists if insurers were to compensate specialist reviewers for their time or if specialists were able to shift their efforts toward higher-value services. In areas of specialist oversupply, the latter strategy may not work because of the difficulty of filling in the lost low-value services through gains in market share. Future work would be needed to address each of these challenges. Knowledge-based methods for collecting more structured data and for intelligent protocols could partially automate the review process, reducing the expense of the specialist review. At the same time, it would be critical to monitor and enhance the satisfaction of PCPs and patients with the referral process, using technology to enhance rather than to degrade the sense of connection to the specialist.