|Home | About | Journals | Submit | Contact Us | Français|
Dr. Singh and Dr. Khan had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.
Given the fragmentation of outpatient care, timely follow-up of abnormal diagnostic test results remains a challenge. We hypothesized that an EMR that facilitates the transmission and availability of critical imaging results through either automated notification (alerting) or direct access to the primary report would eliminate this problem.
We studied critical imaging alert notifications in the outpatient setting of a tertiary care VA facility from November 2007 to June 2008. Tracking software determined whether the alert was acknowledged (i.e. provider opened the message for viewing) within two weeks of transmission; acknowledged alerts were considered read. We reviewed medical records and contacted providers to determine timely follow-up actions (e.g. ordering a follow-up test or consultation) within 4 weeks of transmission. Multivariable logistic regression models accounting for clustering effect by providers analyzed predictors for two outcomes; lack of acknowledgment and lack of timely follow-up.
Of 123,638 studies (including X-rays, CT scans, ultrasounds, MRI and mammography), 1196 (0.97%) images generated alerts; 217 (18.1%) of these were unacknowledged. Alerts had a higher risk of being unacknowledged when ordering providers were trainees (OR, 5.58;95%CI, 2.86-10.89) and when dual (more than one provider alerted) as opposed to single communication was used (OR, 2.02;95%CI, 1.22-3.36). Timely follow-up was lacking in 92 (7.7% of all alerts) and was similar for acknowledged and unacknowledged alerts (7.3% vs. 9.7%;p=0.2). Risk for lack of timely follow-up was higher with dual communication (OR,1.99;95%CI, 1.06-3.48) but lower when additional verbal communication was used by the radiologist (OR, 0.12;95%CI: 0.04-0.38). Nearly all abnormal results lacking timely follow-up at 4 weeks were eventually found to have measurable clinical impact in terms of further diagnostic testing or treatment.
Critical imaging results may not receive timely follow-up actions even when providers receive and read results in an advanced, integrated EMR system. A multidisciplinary approach is needed to improve patient safety in this area.
Communication breakdown is consistently identified as a preventable factor in studies of adverse events1-6 and a significant contributor to outpatient diagnostic errors from lack of follow-up of abnormal test results.7-12 The volume of outpatient care and nature of high-risk transitions between practitioners, settings and systems of care makes timely communication particularly challenging. 13 For example, a patient referred for diagnostic work-up for respiratory symptoms by the primary care provider (PCP) may undergo several laboratory and imaging tests and a pulmonary consultation. Any abnormal findings such as a possible lung mass would need to be communicated rapidly and effectively to the treating providers to ensure adequate follow-up.
Electronic communication using alerts (computerized notifications of critical information such as abnormal diagnostic test results) can facilitate transmission and potentially, a response and follow-up action by an ordering practitioner – an advantage over paper based reporting.14 For instance, the electronic medical record (EMR) used by the Department of Veterans Affairs (VA) mostly relies on a notification system (the “View Alert” window) to alert clinicians about critical test results, whereas only in selected life-threatening cases does the radiologist verbally communicate the abnormality to a provider.14 However, effective communication involves more than just information transfer; it requires a response from the recipient, such as taking a follow-up action and acknowledging receipt of the information to the sender.15 Previous studies raise concern that providers may not read all their alerts.14, 16-19 Data on outpatient drug related alerts has shown providers ignore alerts because of information overload, 20-23 often resulting in failure to act upon important as well as clinically unimportant alerts.20 While no data exists to substantiate a similar problem for diagnostic test result alerts, providers often receive several hundred normal diagnostic test results and many abnormal alerts every week, suggesting that a similar problem could exist here.24
Given that research in this area has thus far focused on critical results that either fail to reach the ordering providers or were somehow “missed” by providers after delivery of information, we hypothesized that an EMR that facilitates the transmission and availability of critical imaging results through either automated notification or direct access of primary report would eliminate this problem. Because there is limited knowledge about why test results get lost to follow-up in EMR systems, we also determined predictors of communication breakdowns (i.e. not reading or taking follow-up action) to help us gain insight about potential interventions to improve safety related to follow-up of abnormal test results. 13, 14
We identified all critically abnormal X-rays, computerized tomography (CT) scans, magnetic resonance imaging(MRI), mammography, and ultrasound alerts transmitted electronically in the multispecialty ambulatory clinic of the Michael E. DeBakey VA Medical Center (MEDVAMC) and its five satellite clinics from November 2007 to June 2008. The local Institutional Review Boards approved the study.
To confirm that providers actually received the alert, we queried the Alert Tracking File, the same repository used by the EMR to populate a provider's View Alert window. We excluded all alerts related to imaging studies done on inpatients. Being their only “inbox” for all types of notifications, providers in the VA are highly dependent on the View Alert system for their abnormal test results. Except for mandatory notifications, receiving various types of notifications is a configurable parameter (for instance, providers have an option to turn normal test result notifications off). We studied mandatory critical imaging alerts transmitted to all types of providers (attending physicians, allied health care providers and trainees) working in a variety of specialties.
The Computerized Patient Record System (CPRS) is the EMR used at all VA facilities. It is maintained within the Veterans Health Information Systems and Technology Architecture (VISTA), an automated information system used to support ambulatory and inpatient care.25 The CPRS includes a notification system (the View Alert system) for alerting clinicians to clinically significant events such as abnormal diagnostic test results. Because of computerized provider order entry, the ordering clinician is always known and notified. When the PCP (usually a permanent staff provider) is out of office, they can assign surrogate covering providers to receive their alerts. When a trainee, a subspecialist, a surrogate, or a substitute provider orders a test, a second alert is automatically transmitted to the PCP (i.e. “dual” alert communication). For the purposes of our study, any provider who received an alert was considered responsible for timely follow-up (e.g. if two providers are alerted we considered them both responsible for timely follow-up).
Using predefined standardized codes, staff radiologists electronically coded abnormal imaging that required action as alerts, 14 which were then transmitted to the View Alert window. The window is displayed when providers log on or switch between patient records and contains alerts on all of their patients, regardless of which record is viewed. Per institutional protocol, alerts stay active in the window for two weeks. Providers are expected to process alerts in a timely manner but have an option of ignoring the View Alert window to bypass it. Occasionally, while evaluating the medical record in response to other reasons, such as a patient visit or phone call, providers may become aware of an abnormal imaging result and review the report without clicking on its corresponding alert.
Two weeks after alert transmission, we queried VISTA through an alert-management-tracking program to determine whether the alert had been “acknowledged,” (i.e. any of the notified providers clicked on and opened the message). If alerts were acknowledged, they disappeared from the View Alert window and we considered them read. Each week, we downloaded both acknowledged and unacknowledged alerts corresponding to our time period (see timeline Figure 1). The downloaded list included patient identifiers and ordering location, names of providers receiving the alert, and the date and type of imaging study. Outpatient alerts where the patient was hospitalized after alert generation but prior to our chart review were categorized separately because the hospitalizations were not always related to the abnormal test result (hence follow-up may not occur). Cases where the radiologist verbally communicated the results (via phone or face to face) to a provider in addition to sending the alert were also categorized separately. Institutional protocol requires that documentation of verbal communication is always included in the imaging report.
A reviewer blinded to acknowledgement status evaluated the medical records of patients to determine if there was any documented response to the alert. Response was defined as any documented evidence of follow-up action, such as contacting the patient or ordering a follow-up test or consultation or hospitalizing the patient for the test result and its presence was considered timely follow-up. If no response was documented, a second study investigator confirmed the findings in the medical record and then contacted the ordering provider by telephone. We used this procedure to determine the provider's awareness of test results and any follow-up action they took but failed to document. If the ordering provider was unavailabe, we called the PCP of the patient (who also receives the alert per protocol). These processes helped confirm whether the test result lacked timely follow-up at the end of 4-weeks after alert transmittal. Lack of timely follow-up was said to occur when there was no response (documented follow-up action at initial chart review) and when the provider had not performed any undocumented follow-up action before our call. We allowed 4 weeks as the maximum time to follow-up on non-life threatening findings based on our previous work.14 When we called providers and lack of timely follow-up was confirmed, we encouraged them to initiate appropriate follow-up action and re-reviewed their records a week later to check for action. Outcome variables were defined as communication breakdown at the electronic acknowledgment step, or the timely follow-up step.
We identified two groups of alerts correponding to the two outcome variables representing communication breakdown in our study: 1) alerts lacking electronic acknowledgment versus acknowledged alerts at 2-weeks after alert transmission, and 2) alerts lacking timely follow-up versus those receiving timely follow-up at 4-weeks after alert transmission. We compared the distribution (as proportions) of several independent variables such as ordering provider specialty (primary care, medicine subspecialties, surgery etc.), ordering provider type (physician, trainee, allied health professionals), type of imaging test (X-rays, CT scans, MRIs, mammography, and ultrasound), use of additional verbal communication by the radiologist and use of dual communication between the groups. Dual communication was counted as one instance of alerting in the denominator. If one of the alerted providers followed up the test, the alert was considered followed-up even if the second provider did not perform any follow-up. The chi-square test was used for categorical variables and Fisher's exact test was used when the assumptions for chi-square were not met. Multivariable logistic regression models accounting for clustering effect by providers were used to identify factors associated with the two outcome variables (i.e. lack of acknowledgment, and lack of timely follow-up). Predictor variables included all covariates with p values <0.2 in univariate testing. The models were fit using maximum likelihood estimation; odds ratios and 95% confidence intervals were calculated. Specialties from univariate analysis were combined into the following groups for multivariable testing: primary care, emergency medicine, surgery and other subspecialties. To ensure the model had stable and precise estimates of odds ratios, variables with small numbers of patients with a given characteristic (such as mammography) were removed, and a final model was run on the remaining predictor variables.
Between November 2007 and June 2008, 123,638 outpatient studies (including X-rays, CT scans, ultrasounds, MRI and mammography) were performed, of which 1196 (0.97%) images generated alerts. Figure 2 illustrates the outcomes of these 1196 abnormal imaging reports. For 206 (17.2%) alerts, the radiologist also communicated verbally with a provider; in 172 (14.4%) the patient was subsequently admitted after alert transmission; and in 73 (6.1%) there was evidence of both. Nine hundred seventy nine alerts (81.9%) were tracked as acknowledged and 217 (18.1%) were tracked as unacknowledged. No evidence of documented follow-up action was found in 131 (11%) of the alerts. In 20 (1.7%) of these cases, we determined calling the provider would be unnecessary or make no impact on outcome either because the diagnosis was not new, the patient was already receiving appropriate care for the condition, or the patient had died. This decision was made after an additional investigator (HS) agreed with the initial reviewer and these alerts were not considered to lack timely follow-up. In the remaining 111 cases, we called the providers to determine if undocumented follow-up care had occurred and if any intentional decision had been made to not pursue follow-up.
Overall, 92 alerts (7.7%) lacked timely follow-up at 4-weeks; thus the estimated rate of lack of timely follow-up was 0.07% of all diagnostic imaging tests performed. In these cases, we re-reviewed the record a week after our initial call to determine if the providers took any follow-up action. We found that in 46 (50%) cases providers had executed the appropriate action.
In Table 1 we compare types of communication, characteristics of ordering providers and types of imaging between the acknowledged and unacknowledged alerts using univariate testing. Additional verbal communication by the radiologist and hospitalization of the patient within 2-weeks of alert transmission was not significantly associated with acknowledgement. Types and specialties of ordering providers and types of imaging tests were significantly different across the two groups. For instance, trainees and surgical specialties were less likely to acknowledge alerts compared to other types and specialties of ordering providers (p<0.0001 for both). X-rays were more likely to be acknowledged (p=0.0005) than other imaging tests.
In a logistic regression model for lack of acknowledgment, the following variables were significant (data not shown in tables): physician assistants as ordering providers as compared to attending physicians (OR: 0.46; 95%CI: 0.22-0.98), trainees as ordering providers (OR: 5.58; 95%CI: 2.86-10.89), and when dual as opposed to single communication was used (OR: 2.02; 95%CI: 1.22-3.36).
We further evaluated trainee unacknowledgment. Of 74 alerts unacknowledged by trainees that were followed by a documented response in the medical record at the time of review, the ordering trainee directly responded to the test result (without opening the alert) in 34 cases (9 of which were prompted by a direct notification by the radiologist). In the remaining 40 cases, another provider documented a response to the report (4 of which were prompted by a direct notification by the radiologist).
Table 2 shows a univariate analysis comparing the following characteristics between the 92 alerts determined to lack timely follow-up at 4-weeks versus those that received timely follow-up: presence or absence of acknowledgment, types of communication, characteristics of ordering providers, and types of imaging. There was no significant difference in rates of lack of timely follow-up between the acknowledged and unacknowledged alerts (7.3% vs. 9.7%; p=0.2). Additional verbal communication and instances where the patient was hospitalized within 2 weeks of alert transmission were significantly more likely to be associated with timely follow-up (p< 0.0001). No statistically significant differences in provider type or specialty was seen, but certain tests (CT scans and MRIs) were more likely associated with timely follow-up (p= 0.002).
In a logistic regression model for lack of timely follow-up, the following variables were significant predictors (Table 3): additional verbal communication (OR: 0.12; 95%CI: 0.04-0.38), a subsequent hospital admission after alert transmission (OR: 0.22; 95%CI: 0.08-0.38), alerts of CT scans (OR: 0.49; 95%CI: 0.29-0.85) and when dual as opposed to single communication was used (OR: 1.99; 95%CI: 1.06-3.48).
Table 4 shows the types of near-miss abnormalities [defined as events that could have harmed the patient but did not cause harm as a result of chance, prevention, or mitigation26] in cases that lacked timely follow-up versus those that did not. Chest imaging showing a non-specific density was more likely to be associated with lack of timely follow-up (p=0.04). All near-misses had potential to lead to missed or delayed diagnosis had we not intervened; the majority of these (62 of 92; 67.4%) were related to some form of a suspected new malignancy. Half (50.0%) of the tests lacking timely follow-up were abnormal chest x-rays. Although obtaining qualitative data from practitioners was not a part of this study, seven providers we called volunteered that someone else other than them (e.g. the PCP) was the provider responsible for follow-up.
Subsequent evaluation of the importance and outcomes of the 92 imaging tests that did not receive timely follow-up was conducted, collecting data on outcomes that were the direct result of the imaging finding in question. Table 5 lists the outcomes determined at 9 months after the date of the test. Although in 5 cases we could not determine outcomes because the patient did not return to the institution, all other tests had a measurable clinical impact in terms of either diagnostic testing or treatment. In these 92 cases, more than a fourth (n=26; 28%) led to a new diagnosis being established. Cancer was determined to be a new diagnosis in 11 of these cases.
In a study of 1196 critical imaging notifications in an integrated EMR, we found 92 (7.7%) results without timely follow-up at 4-weeks after result transmission. Our findings suggest that an EMR that facilitates transmission and availability of critical imaging results to the provider through either automated notification or direct access of primary report does not eliminate the problem of missed test results even when one or more providers read the results. Communicating alerts to two recipients, which occurred when tests were ordered by personnel other than the regular PCP, significantly increased the odds that the alert would not be read and would not receive timely follow-up action. This could have resulted because the lines of responsibility for follow-up were less clear with dual communication; the PCP may assume the ordering provider was providing follow-up and vice-versa. Verbal communication by the radiologist in addition to the electronic communication strongly predicted timely response and follow-up, probably because radiologists called only for emergent and life-threatening findings. Nearly all abnormal results lacking timely follow-up were eventually found to have measurable clinical impact in terms of further diagnostic testing or treatment.
We unexpectedly found that imaging tests flagged “critical” even when confirmed to have been read by providers may not receive timely follow-up actions. Therefore, even in the best of information systems that contain advanced notification features, patients with abnormal imaging tests are vulnerable to “fall through the cracks”. This underscores the need for a multi-disciplinary approach involving human and computer interaction and informatics.27, 28 to complement the benefits achieved by automated notification and the need for continuous monitoring procedures to ensure follow-up even when providers “acknowledge” i.e. read abnormal results. The impact of several factors such as time pressures, organizational characteristics, factors related to the task of processing alerts, technological factors and provider factors should be considered in the design of interventions to improve timely follow-up.29, 30 Future studies should address process-of-care issues leading to such communication breakdowns and guide the design and implementation of the next generation of computerized notification systems in ambulatory care.
Our study provides a new and more accurate estimate of prevalence of critical imaging test results without follow-up within a defined time interval. In our previous work, we calculated this to be about 4% of critical imaging results, but this figure was based only on unacknowledged alerts.14 Unexpectedly, our current study shows substantial lack of timely follow-up even in alerts read by providers; hence the increase from our previous estimate. However, we believe that our estimates of lack of timely follow-up are likely better than in systems that do not use computerized notification, although accurate and comparable data from such systems is lacking.9, 31
Trainees had a higher likelihood of not reading their alerts but were not associated with lack of timely follow-up. In our system, as in many others, residents spend only ½ to 1 day in a two-week period in an outpatient setting usually in their own continuity clinics. In addition, duty hour rules also impact their clinic schedule. Therefore, an alert related to an abnormal imaging test ordered by a resident may be not be subsequently opened and read by the same resident in a timely manner. Rather, an alternate or supervising physician may either receive a call from the radiologist or access the primary report in response to an alert, in their absence. In more than half of alerts unacknowledged by residents where a documented response was found in the medical record, it was another provider which documented that response (i.e. alternate or supervising physician).
Several predictors provided important information for future work in improving patient safety in this area. First, dual communication (defined as when an alert was transmitted to two providers), intended to be a “safe-guard” to protect against loss of follow-up was unexpectedly associated with lack of timely follow-up. Preliminary evidence from our study suggests providers may believe responsibility for follow-up belonged to someone else. While institutional practices may determine whether the PCP or another ordering provider (for instance the subspecialist) follows up an abnormality, this problem may be improved by better tracking systems,14, 32-34 using reminders built into the EMR, and care-coordination programs.27, 35-38 Second, direct (verbal) communication is important and will continue to play a valuable role even in sophisticated EMR systems. Currently in our institution direct communication is required only when tests results are life-threatening or require an emergent intervention. However, our findings suggest that providers may perceive a lack of urgency for test results that have less immediate consequential implications (e.g. suspicious for a new cancer). Policies and procedures regarding the use of verbal communication for such results may need to be re-evaluated. 39, 40
While the next generation of EMR based notification systems are being designed, we propose several potential interventions based on our findings that can be used immediately to improve timely follow-up of abnormal imaging results. First, every institution must develop and publicize a policy regarding who is responsible (PCP vs. the ordering provider, who may be a consultant) for taking action on abnormal results. Second, unacknowledged alerts must stay active on the EMR screen for longer periods, perhaps even indefinitely, and should require the provider's signature and statement of action before they are allowed to drop off from the screen. This would also ensure alerts do not “disappear” before the provider has a chance to order the follow-up action. Third, interventions to reduce alert overload and improve signal to noise ratio of alerts should be explored. For instance, unnecessary alerts should be minimized and alerts should be retracted when the patients dies or if the radiologist calls, or the patient is admitted before the alert is acknowledged.41 Fourth, the EMR should be automated to track not just acknowledgement but the specific response documenting that appropriate action has been taken for each abnormal alert. This would save time and expense of performing chart reviews and calling providers to confirm their actions. Lastly, an audit and performance feedback system should be established to give providers information on timely follow-up of patients’ test results on a regular basis.42
Our study has several limitations. Because sophisticated tracking such as the one we used is not easily possible in the non-EMR environment and similar data from the pre-EMR era in the VA does not exist, lack of a comparison group was a limitation. Because of the study population and site our findings may not be generalizable outside the VA setting; especially to free standing clinics that do not use integrated EMR systems. However, with recent emphasis of EMR adoption and implementation nationally, we believe that our findings are of great significance. In addition, we did not account for any problems with alert transmission failures such as situations when the radiologist inadvertently did not code the report as an abnormality. Conversely, many factors including a large sample size, involvement of several clinics, large number of patients (over 120,000 veterans in southeast Texas) and providers (over 500 from different specialties), rigorous methods to determine timely follow-up and outcomes at 9 months of alerts without timely follow-up, various types of diagnostic imaging tests and an advanced integrated EMR used in VA facilities nationwide, all add unique strength to our study.
An EMR that facilitates transmission and availability of critical imaging results to the provider through either automated notification or direct access of primary report does not eliminate the problem of missed test results even when one or more providers read the results. Dual communication, intended to be a “safe-guard” to protect against loss of follow-up, was unexpectedly associated with lack of timely follow-up. Future multidisciplinary studies should address process-of-care issues leading to such communication breakdowns and guide the design and implementation of the next generation of computerized notification systems in ambulatory care.
We acknowledge Dr. Meena Vij for her support for our project.
The study was supported by an NIH K23 career development award (K23CA125585) to Dr. Singh, the VA National Center of Patient Safety, Agency for Health Care Research and Quality and in part by the Houston VA HSR&D Center of Excellence (HFP90-020).
These sources had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; and preparation, review, or approval of the manuscript.
The views expressed in this article are those of the authors and do not necessarily represent the views of the Department of Veterans Affairs.
Conflicts of Interest