We previously demonstrated that implementation of an integrated package of quality improvement interventions that utilizes decision support, a provider education (activation) campaign, feedback reports and organizational changes more than doubled HIV testing rates for at-risk individuals18
. These results were robust with dramatic increases in the likelihood of being tested for HIV being observed across patient-level, provider-level and subfacility-level factors. Furthermore, the fraction of HIV test results that were positive remained constant (0.45%) and well within the range at which HIV testing costs less than $50,000 per quality-adjusted life year when societal benefits of testing are considered6
We now report on the sustainability of this program during the twelve-month period after overall responsibility for the interventional program was transferred to preexisting clinical management, who chose to greatly deintensify the provider education campaign and other labor and time-intensive aspects of the intervention18,35
. Remarkably, we found that the rate of HIV testing continued to increase for patients making their first, second or third visits during the sustainability period. These results indicate that despite the de-emphasis of the provider education campaign, when the frequency of medical contact is considered, the program’s impact on HIV testing rates was fully sustainable. The observation that overall testing rates declined was related to the changing make-up of the study population as patients with their first through third visits accounted for 100% of the study population in month 1, 54% of the population in month 12 and 41% of the population in month 24.
We also found that the rate at which patients refused HIV testing decreased over time. Correspondingly, the likelihood of having the HIV Testing Clinical Reminder being resolved by HIV testing increased. These results suggest that providers became more proficient at offering and discussing HIV tests and may have integrated HIV testing into their normal practice. Others have observed that normalization of HIV testing is associated with increased patient acceptance of testing36,37
The importance of reporting the sustainability of health care interventions and of choosing appropriate measurement metrics is receiving increasing attention32
. Our results indicate that assessments of the sustainability of the outcome of an intervention are critically dependent on the mode of analysis. We found that when applied to homogeneous patient population (as defined by prior use of VA healthcare), increased HIV testing rates were sustained after de-emphasis of the provider education campaign and continued to increase among patients newly exposed to the intervention (Fig. a). This suggests that our intervention has become part of the institutional culture of our facility, does not overburden providers and fits the implementing culture and variations of the patient population32
Stratified analysis by the number of visits during each year reveals that our intervention was least sustained among established patients who had not previously been offered testing. We conclude that interventions that aim to maximize sustainability should consider a “tail” of provider education or other components focused on patients who do not receive recommended services on the first exposure. Also, further work needs to be done to determine the determinants of repeated non-performance. We believe that such failures are likely due to systemic barriers or a lack of provider agreement/knowledge. Notably, although theoretical38–40
and empirical observations22,23,41–43
demonstrate that the use of provider education (or activation) campaigns are necessary to transform group norms and maximize quality improvement, there is far less literature regarding the importance of maintaining these activities to sustain whatever gains are achieved during their use32
The strengths of our sustainability analysis include, as recommended, use of a time-series analysis of monthly rates of HIV testing which allowed us to better assess the trajectory of HIV testing rates32,44
. Furthermore, we examined the effectiveness of the intervention in an unselected population of at-risk veterans receiving care in a routine, real-world clinical setting.
Limitations include the fact that the sustainability analysis was done immediately after the withdrawal of study personnel from active maintenance of the intervention. It is therefore difficult to distinguish between lingering improvements from the implementation and true persistence of effects from institutionalization45
. Moreover, this study was undertaken within the quality improvement infrastructure in the VA, which includes an electronic medical record, clinical reminder software and familiarity with performance measurements. Although such tools are increasingly common, this intervention might not be generalizable to other healthcare systems. Another limitation is that while sustainability can be defined as continued use of the core elements of the interventions, and persistence of improved performance32
, we did not formally evaluate the continued use of the core elements of the interventions or their individual contributions to the successful sustenance of the intervention. However, surveys of the two HCSs involved in this project indicate that the organizational changes that favor HIV testing and the HIV Testing Clinical Software package have been maintained. Another limitation is that there was still room for improvement and it is unknown whether the rates of HIV testing would have increased further had the provider activation campaign been continued. Furthermore, while guidelines now recommend that all patients be offered HIV testing and that yearly testing be offered to persons who continue to engage in high risk activities14,46–48
, this intervention was targeted to ensure one-time testing in patients with known risk factors. This strategy was purposely undertaken to prioritize testing for patients at the highest known risk for HIV infection and in deference to concerns that a program to promote HIV testing in all patients would be impractical in the VA as long as written informed consent was required for testing. Finally, the achieved rate of HIV testing remained less than desired. It will be important to determine the effect of removal of the written informed consent requirement for VA HIV testing in August 2009 on the rates of HIV testing49
In conclusion, we found that when assessed in homogeneous patient populations, the impact of implementation of the coordinated use of a computerized clinical reminder, feedback reports, provider education and organizational change is sustainable after cessation of external support of the provider education component. Maintenance of the gains after withdrawal of support by the research team suggests that the organizational and behavioral changes that led to the enhanced performance of HIV testing were successfully institutionalized. These findings have substantial implications for the assessment and sustenance of quality improvements programs for clinical preventive services and beyond.