|Home | About | Journals | Submit | Contact Us | Français|
On March 11, 2009, the Veterans Health Administration (VA) implemented an electronic health record (EHR)-based intervention that required all pathology results to be transmitted to ordering providers via mandatory automated notifications. We examined the impact of this intervention on improving follow-up of abnormal outpatient pathology results.
We extracted pathology reports from the EHR of two VA sites. From 16,738 pre- and 17,305 post-intervention reports between 09/01/2008 and 09/30/2009, we randomly selected about 5% and evaluated follow-up outcomes using a standardized chart review instrument. Documented responses to the alerted report (e.g., ordering follow-up tests or referrals, notifying patients, and prescribing/changing treatment) were recorded.
Primary outcome measures included proportion of timely follow-up responses (within 30 days) and median time to direct response for abnormal reports.
Of 816 pre- and 798 post-intervention reports reviewed, 666 (81.6%) and 688 (86.2%) were abnormal. Overall, there was no apparent intervention effect on timely follow-up (69% vs. 67.1%;p=0.4) or median time to direct response (8 days vs. 8 days; p=0.7). However, logistic regression uncovered a significant intervention effect (pre-intervention OR, 0.7; 95%CI 0.5-1.0) after accounting for site-specific differences in follow-up, with a lower likelihood of timely follow-up at one site (OR,0.4; 95%CI 0.2-0.7).
An electronic intervention to improve test result follow-up at two VA institutions using the same EHR was found effective only after accounting for certain local contextual factors. Aggregating the effect of EHR interventions across different institutions and EHRs without controlling for contextual factors might underestimate their potential benefits.
Lack of follow-up of test results is a widely-prevalent safety concern.(1-4) While delays in follow-up of anatomic pathology results have been described in cancer-related malpractice claims,(5) prevalence of this problem is unknown. The use of electronic health records (EHRs) can potentially overcome these concerns. However, electronic solutions to facilitate communication have had varied success(1,6-9) and few studies have explored the complex intersection between the pathology laboratory and the clinical microsystem.(10)
In the Veterans Health Administration (VA), test results are communicated from the laboratory to providers through automated messages in a “View Alerts” inbox within the EHR. (11) Alerts are delivered asynchronously (such as in email, when message transmitter and receiver are not simultaneously engaged) and are distinct from the synchronous order-check alerts dynamically created during order entry, for example, to warn providers of potential drug-interactions. Asynchronous alerts of this type are “passive” alerts and do not directly interrupt workflow or necessitate an immediate response. They are not unique to the VA; various EHR systems have comparable tools for notification.(12-13)
Across VA sites, local policies and committees decide which types of test result alerts are “mandatory” (notifications that cannot be switched off by providers).(11) Anatomic pathology deals with the diagnosis of disease based on gross, microscopic and molecular examination of tissues. Until recently, these results were not mandatory. Although providers can customize their alert settings to receive some additional non-mandatory notifications, many providers only use locally-set default options.(11). In response to concerns that important findings might be missed in such scenarios, a national intervention was implemented throughout the VA on March 11, 2009, requiring all pathology results (normal or abnormal) to be transmitted to ordering providers as mandatory alerts (i.e. automatically generated notification for every result). We examined the impact of this intervention on follow-up of abnormal pathology results in the outpatient setting of two VA sites. We hypothesized that timely follow-up will improve post-intervention at both institutions by decreasing time to follow-up and eliminating missed abnormal results.
We evaluated test result follow-up outcomes through retrospective chart reviews six-months pre-and post-intervention. The project was IRB approved.
We selected two geographically disparate VA sites (Sites A and B) where the local default for anatomic pathology alerts was non-mandatory to the ordering providers prior to March 2009. Sites were chosen based on research team’s convenience and feasibility of data extraction and review. Site A is larger; both are teaching, tertiary-care referral centers with multispecialty clinics and community-based satellite clinics that provide care to urban and rural veterans.
At both sites, we reviewed the EHR and collected data on normal and abnormal outpatient pathology results reported between 09/01/2008 and 02/27/2009 pre-intervention and between 04/01/2009 and 09/30/2009 post-intervention. The total number of reports generated during the study period was comparable at these sites (Site A: 8497 pre-intervention and 8839 post-intervention; Site B: 8241 pre-intervention and 8466 post-intervention).
We queried the EHR database to generate a list of outpatient pathology reports that contained information on patient-identifiers, ordering-provider name, type of pathology report (surgical pathology or cytology reports) and date. Other descriptive variables (ordering-provider type, specialty and final results) and follow-up outcomes were collected through manual record-reviews.
We identified 16,738 pre-intervention and 17,305 post-intervention reports at both sites. In absence of prior data, we estimated our review sample size based on the intervention leading to a significant reduction of time to follow-up action by 5 days (from average follow-up time of 12 days for abnormal results based on pilot reviews to an anticipated 7 days). This suggested a minimum sample size for each group to be 284 abnormal tests per site (power of 0.80, alpha=0.05, total abnormal tests =568). Based on an abnormal to normal test ratio of 7:3, we randomly selected records on a monthly basis, oversampling by 5 records at each site to account for any inappropriate records (i.e. inpatient records miscoded as outpatient, duplicate reports, etc.). Thus we randomly selected just over 800 total reports for chart review pre- and post-intervention.
At each site, a trained physician-reviewer collected data from the EHR using a standardized data collection instrument. Reviewers were trained during pilot testing to ensure consistent data collection on follow-up actions.(8-9,14) Definitions of follow-up actions and the respective examples are listed in Table 1. Lack of follow-up was defined as absence of direct response to the test and indirect follow-up actions in situations where follow-up was required.
Primary outcome measures (i.e. most sensitive to intervention) included:
Pre-and post-intervention secondary outcome measures included:
In addition to generating descriptive data, we compared the distribution (as proportions) of several independent variables within each group including ordering provider specialty (primary care, medical or surgical subspecialties) and ordering provider type. Each individual report was considered the unit of analysis because each required a unique action.(15) Association of dichotomous or categorical variables was accomplished using the chi-squared test. A two-sided p-value was used to test for significance in all cases. Statistical significance was defined using a criterion of P<0.05. Each variable was tested individually for inclusion in the overall logistic model using logistic regression with the variable as the only covariate. All variables p < 0.25 were included as candidates. A generalized estimating equation (GEE) model was used for the overall logistic regression to account for patients being nested within providers. (16) All analyses were performed using R statistical software version 2.10.1 and SAS 9.2.
Of 830 pre- and 807 post-intervention reports reviewed, 23 charts with missing data were excluded from analysis; abnormal reports included 666 of 816(81.6%) and 688 of 798(86.2%) respectively. Post-intervention, there were more general medicine practitioners [227/798(28.5%) vs. 176/816(21.6%)] and fewer dermatologists [135/798(16.9%) vs. 174/816(21.3%)] and pulmonologists [14/798(1.8%) vs. 32/816(3.8%)] in our sample. There were more Papanicolaou smears [257/798(32.2%) vs. 211/816(25.9%)] and fewer shave biopsies [101/798(12.7%) vs. 139/816(17.0%)] performed post-intervention in our sample. Trainees accounted for about half of the ordering providers [382/798(47.9%) post-intervention vs. 427/816(52.3%) pre-intervention].
We further compared characteristics of pathology reports in pre- and post-intervention groups at each site (Table 2). There were proportionally less trainees and nurse practitioners sampled in Site A. More malignant lesions were found in reports sampled from Site A[14.1% vs. 7.8% pre- and 11.4% vs. 6.4% post-intervention].
Overall, timely follow-up for abnormal reports was not significantly changed post-intervention; [447/666(67.1%) pre- versus 477/688 (69.3%) post-intervention (p=0.4)]. There was no significant change in rate of timely follow-up at the two sites, although Site B had higher rates of timely follow-up [104/117(88.9%) pre- and 111/117(94.9%) post-intervention, p=0.09] compared with Site A [343/549 (62.5%) pre- and. 366/571(64.1%) post-intervention, p=0.6]. The median time to direct response was unchanged post-intervention [8 days (inter-quartile range (IQR) 5-18 days) vs. 8 days IQR (5-15 days) respectively; p=0.65]. Individually, it was unchanged at 15 days for Site A, but decreased slightly at site B from 9 to 7 days (p>0.05).
Table 3 shows outcomes in terms of follow-up. Direct responses to abnormal reports were unchanged post-intervention (p=0.3).
Lack of follow-up for abnormal reports at 6 months decreased post-intervention [10.1% pre- vs. 3.1% post-; p<0.05] (Table 3). Site A accounted for nearly all reports without follow-up [11.8% pre- vs. 4.2% post-;p<0.05]. Overall, documentation of patient notification for abnormal reports decreased slightly post-intervention [423/666 (63.5%) pre- vs. 411/688 (59.7%) post-intervention; data not shown]. Individually, documentation at Site B was higher [100/117(85.5%) pre- and 97/117(82.9%) post-intervention vs. 323/549(58.8%) pre-and 314/571(54.9%) post-intervention; data not shown].
In a logistic regression model for timely follow-up (Table 4), an intervention effect was demonstrated; the pre-intervention group was less likely to receive timely follow-up (OR,0.7; 95% CI 0.5-1.0). Site-specific differences existed; Site A was less likely to provide timely follow-up (OR,0.4; 95%CI 0.2-0.7), even after accounting for differences in provider and test report characteristics The following specialties were significantly more likely to be associated with timely follow-up after accounting for the possible intervention and site effect: Hematology/Oncology (OR, 8.7;95%CI 1.3-57.5), Pulmonology (OR,24.4;95%CI 3.3-181.2), and Urology (OR,5.3;95%CI 1.8-15.6); Older patients (>80 years were more likely to receive timely follow-up (OR,1.6; 95%CI 1.1-2.4).
An EHR-based “mandatory” notification of anatomic pathology results improved the proportion of patients who received follow-up at six months. However, an intervention effect on timely follow-up was shown only after accounting for various site, provider and test variables in a logistic regression model. After controlling for facility differences, certain types of specialists and older patients were more likely and trainees were less likely to be associated with timely follow-up. Follow-up was remarkably different in the two study sites despite the use of the same EHR. This likely reflected differences in local practices and workflow features which we are unable to capture using chart review.(17) Our findings suggest that technology-based interventions to improve test results management in different organizations are likely to exert a highly variable “real-world” effect even when health care systems and technology are similar.
To our knowledge, this is the first study to establish rates of follow-up of anatomic pathology results in the setting of an integrated EHR. Our study also has significant implications for EHR-based interventions targeting effective communication of test results. Despite the same intervention in the same EHR, the intervention had no impact on the pre-existing differences in follow-up patterns between the two sites. Implementation and use of health information technologies in complex systems requires addressing many contextual factors beyond technology for achieving their effectiveness.(18-25) Local “socio-technical” factors such as existing workflows or practices, concomitant quality improvement initiatives and other context factors (personnel and organizational features etc.) must be taken into account.(26)
Although further qualitative work is essential to fully understand our findings, several contextual factors could likely explain these differences.(27) For instance, there are few standardized clinical practices or workflows for fail-safe management of test results and the level of institutional support providers receive for test result management activities is variable. Individual provider factors, related to how they manage test results in the EHR, might be especially prominent and need to be explored further.(11) Some providers might not have been able to access alerts. For instance, certain specialists and trainees who rotate within the VA might not remotely access the EHR. Currently, these alerts reliably only go to a single person (i.e. the ordering provider), who might be off-site. Site-specific differences in management of alerts sent to trainees may exist, but test result follow-up by trainees was still untimely after controlling for these differences. Additionally, many providers can receive over 50 different types of notifications a day (28) and due to a large number of notifications, a “needle in haystack” phenomenon might result where abnormal pathology reports may be under-prioritized or overlooked.(29) This might explain why general medicine providers, who typically receive more alerts, were less likely to provide timely follow-up than sub-specialists.
Our study limitations include a lack of control group for comparison to account for temporal trends. This was not feasible because this was a natural experiment throughout the VA. Improvements may occur beyond six months post-intervention, which we did not measure. While our study findings might not be considered generalizable beyond the VA, many EHRs are adopting notification systems similar to the VA and our lessons could be useful for them. Finally, we relied on EHR documentation to determine outcomes and might have missed actions not documented. However, if at all, documentation should have been higher post-intervention because a VA directive co-incidentally also implemented in March 2009 required all test results to be communicated to patients within 14 days of result and for this communication to be documented in the EHR.(15)
In conclusion, our study suggests that aggregating the effect of EHR interventions across different institutions and EHRs without controlling for local “socio-technical” contextual factors might underestimate their potential benefits.
Funding Source: The study was supported by an NIH K23 career development award (K23CA125585) to Dr. Singh, the VA National Center of Patient Safety, Agency for Health Care Research and Quality and in part by the Houston VA HSR&D Center of Excellence (HFP90-020). These sources had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; and preparation, review, or approval of the manuscript.
The study was supported by an NIH K23 career development award (K23CA125585) to Dr. Singh, the VA National Center of Patient Safety, Agency for Health Care Research and Quality and in part by the Houston VA HSR&D Center of Excellence (HFP90-020).
These sources had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; and preparation, review, or approval of the manuscript.
Conflicts of Interest
Dr. Laxmisan and Dr. Pietz had full access to all of the data in the study and take responsibility for the integrity of the data and the accuracy of the data analysis.
This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.