|Home | About | Journals | Submit | Contact Us | Français|
Among the expected benefits of electronic health records (EHRs) is increased reporting of public health information, such as immunization status. State and local immunization registries aid control of vaccine-preventable diseases and help offset fragmentation in healthcare, but reporting is often slow and incomplete. The Primary Care Information Project (PCIP), an initiative of the NYC Department of Health and Mental Hygiene, has implemented EHRs with immunization reporting capability in community settings.
To evaluate the effect of automated reporting via an EHR on use and efficiency of reporting to the NY Citywide Immunization Registry, we conducted a secondary analysis of 1.7 million de-identified records submitted between January 2007 and June 2011 by 217 primary care practices enrolled in PCIP, pre and post launch of automated reporting via an EHR. We examined differences in records submitted per day, lag time, and documentation of eligibility for subsidized vaccines.
Mean submissions per day did not change. Automated submissions of new and historical records increased by 18% and 98% respectively. Submissions within 14 days increased from 84% to 87%, and within 2 days increased from 60% to 77%. Median lag time decreased from 13 to 10 days. Documentation of eligibility decreased. Results are significant at p<0.001.
Significant improvements in registry use and efficiency of reporting were found after launch of automated reporting via an EHR. A decrease in eligibility documentation was attributed to EHR workflow. The limitations to comprehensive evaluation found in these data, which were extracted from a registry initiated prior to widespread EHR implementation suggests that reliable evaluation of immunization reporting via the EHR may require modifications to legacy registry databases.
Immunization registries are confidential, population-based, computerized databases that record all immunization doses administered by participating providers to persons residing within a given geopolitical area [1, 2]. State and local registries offset fragmentation in healthcare by consolidating scattered records, by reducing unnecessary vaccination among patients who have lost records or visit multiple providers, and by tracking vaccine formulations, supply, and schedules . In 2010, the Task Force on Community Preventive Services recommended registries to increase vaccination coverage based on strong evidence of their effectiveness for managing immunizations . Registry data also are used to evaluate vaccine programs and population coverage [5-10]. However, accuracy, timeliness and completeness of data submitted are highly variable [9, 11-22], reducing the impact of these registries. Electronic health records (EHRs) provide a potential solution when immunizations documented at the point of care are reported automatically, directly to the registry [5, 23].
In 1997 the Citywide Immunization Registry (CIR) was established as a centralized repository of immunization records for people vaccinated in NYC. Over 2,000 medical practices and other venues are mandated to report immunizations administered to children 19 and older who reside in NYC. [19, 24]. There are three ways to submit immunization records to CIR:
Starting in 2007 EHRs equipped with an automated immunization reporting module were implemented in community practice settings by the Primary Care Information Project (PCIP). This initiative of the NYC Department of Health and Mental Hygiene subsidized adoption of EHRs, targeting practices serving communities at risk for health disparities determined by Medicaid visits per zip code . Within the EHR a customized application, or module, automates the process of reporting immunization records via an Internet connection to CIR established when a practice first deploys the module. Each evening the module automatically aggregates the day’s records of immunizations administered and uploads a batch file directly to the CIR Web File Registry (WFR) . This nightly process extracts immunization data directly from the native EHR database with no additional workflow for staff. The specifications for the module were developed collaboratively by PCIP and CIR and implemented by the vendor eClinicalWorks (eCW). At the time of the study immunization reports used a legacy reporting format, and CIR was planning transition to the national HL7 standard. To date, there are approximately 300 practices reporting to CIR via an automated immunization module.
We conducted a secondary analysis of immunization records submitted to CIR by a cohort of these practices to evaluate the effect of automated EHR-based reporting on use of the registry and efficiency of reporting. To our knowledge there are no previous studies of registry use and efficiency related to EHR-based reporting.
Practices were eligible if they were enrolled in PCIP, had implemented the immunization module in their EHR, and were certified by CIR to use the module to submit immunization records. In general, a practice is certified after approximately 30 days of rigorous system testing by CIR staff.
We obtained a de-identified, limited data set of vaccination records submitted to CIR by practices that met the study criteria. The data covered children ages 19 and under and represented all immunizations, including influenza, submitted to the CIR between January 1, 2007 and May 31, 2011. Records submitted in the 1.25 years before the CIR certification date for each practice were considered pre-implementation records. Records submitted in the 1.25 years after were considered post-implementation records. This time period was selected to reflect a full year of reporting before and after certification, including additional time (0.25 years) for transition pre/post. In all cases daily submission counts above the 99th percentile were removed to minimize skew in the distribution. MySQL was used to extract relevant data sets for analysis. SAS was used for statistical tests and to generate regression plots. The study was approved by Columbia University Institutional Review Board.
To measure effect on use of the registry we tested pre and post implementation differences in the mean number of records submitted per day via all sources, via the online registry, and via automatic file transfer. In the sub set of records submitted via automatic file transfer we examined new records (vaccines submitted ≤ 1 year of delivery) and historical records (submitted >1 year post delivery, or flagged as historical by the provider). These data were normally distributed and the student’s t-test was performed.
To evaluate the efficiency of reporting we excluded historical records from the analysis to eliminate the effect of large batches of backlogged records submitted by individual practices shortly after EHR immunization module deployment. We tested pre and post implementation differences in the median percentage of records from all sources uploaded within the 14-day requirement and within 2 days. Due to the nightly record upload a 2-day window represents immediate reporting via the module and reflects an increased use of the module. The 14-day reporting period is the legal reporting requirement and represents a broader measure of reporting efficiency by study practices. We tested pre and post implementation lag time, defined as the difference in days between the vaccine administration date and the date it was reported to the CIR. To approximate record completeness pre and post implementation we determined the median percentage of records submitted from all sources which had a non-null value in the field indicating eligibility for the Vaccine for Children program (VFC), a federal program that subsidizes vaccines. CIR tracks vaccines administered through this program. These data were asymmetrically distributed and the Wilcoxon rank sum test was performed. We also performed segmented regressions to allow visual inspection of pre/post data plots and to test for significant differences in slope.
There were 217 practices that met the study criteria. Of these 82% (n = 178) had submitted records to CIR before implementation of automated reporting via the EHR. About 18% (n = 39) began submitting to CIR after implementation, despite the NYC reporting requirement. The data obtained consisted of 1.7 million immunization records representing 220,000 unique individuals. It should be noted that the population of children was stable during the study period, as shown in ► Table 1 .
Although the mean number of records submitted daily to CIR from all sources remained constant, the pattern of submissions changed significantly. The results are shown in ► Table 2.
Submissions via the online registry decreased from a mean of 1061 to 369 per day. Submissions via automated file transfer increased from mean of 1451 to 2051 per day. For new records automated file transfers increased from a daily mean of 917 to 1085 and for historical records from 497 to 985. Median lag time before implementation was 13.3 days [IQR = 10.7, 17.2], and after was 9.8 days [IQR = 6.4, 15.9]. Median lag time for manual submissions before implementation was 32.7 days [IQR = 25.9, 39.9], and after was 80.3 days [IQR = 61.6, 105.6]. The percentage of records submitted to CIR within the 14-day legal limit improved from 84% to 87%. The percentage of records submitted within 2 days increased from 60% to 77%. The proportion of records with the Vaccine for Children field completed decreased significantly after implementation, from 70% to 62%. These differences are significant at p<0.001.
A segmented regression plot in ► Figure 1 shows the percent of immunization records reported within 2 days of vaccine administration. The certification dates of each practice were aligned and outliers above the 99th percentile were removed. There is a visually notable decrease in dispersion of submissions after automated reporting with a significant pre/post difference in slope of the regression line (p<0.001).
The combined evidence suggests that automated reporting via the EHR has facilitated registry use for these 217 practices. The overall mean number of records submitted per day remained stable, suggesting already high compliance by providers with reporting mandates as well as suggesting a stable number of immunizations during the study. Automatic submissions (both new and historical) did increase in proportion to all submissions, and records from 39 previously non-compliant practices were captured. The increases in the proportion of submissions within 2 and within 14 days may underestimate the true efficiencies achieved since 18% of practices had not previously reported. The decreases in 2 and 14 day lag time are corroborated by the decreased data dispersion visualized post certification.
We did not assess how automated reporting affected workflow, and cannot speak to whether the gains in efficiency for CIR came at the expense of any clinician time required to use the reporting module. We might argue the greater lag in manual reporting that we found post implementation suggests a preference for automated reporting, but it could also suggest over reliance on automated reporting, which would represent an unintended consequence. As always, further research is needed to fully comprehend the effects of this, or any, implementation.
The decline in completeness of the VFC eligibility field may be due to multiple factors. Records submitted via the EHR did not contain a field for Medicaid numbers, a criterion from which eligibility may be determined. Further, a provider could document VFC eligibility in more than one field, but automated file transfer captured only one of those fields, suggesting that the EHR workflow may present difficulties for recording eligibility. CIR staff is evaluating these issues.
The study was greatly limited by the registry database structure: records submitted electronically via a non-EHR practice management system versus the EHR immunization module could not be dis-aggregated. This fact limited our analysis to pre- and post-certification of electronic reporting aggregated at the practice level. We were unable to discern if practices may have used non-EHR systems after certification to submit records electronically. We believe that situation is unlikely for two reasons: reporting via the EHR was automated to deliver records in a more timely way, and our findings clearly documented such a change. The data structure also limited our ability to ascertain a rate of immunization based on the patient population of the study practices. Significant changes in the underlying patient population could impact the interpretation of the study results. We don’t believe this to be the case as the the overall patient population was stable during the study period (► Table 1) and no significant changes in the mean immunizations reporting pre and post implementation were found (► Table 2). The structural limitation presents a cautionary lesson for those considering analysis of data from legacy immunization registries to evaluate reporting via the EHR. Perhaps our strongest finding is to illustrate that evaluation of the impact of automated immunization reporting will require the collection and reporting of additional data elements by EHR systems and registries to improve the accuracy and reliability of future studies.
Immunization recommendations in the US currently target 17 vaccine-preventable diseases across the lifespan . As EHRs proliferate, automated reporting of immunizations can help public health officials and individual providers better determine who in the patient population has been adequately immunized, reduce provider paperwork and staff time, provide easy access and reliable immunization histories, and reinforce the concept of the medical home . Improved efficiency of immunization registry reporting via an EHR is of increased importance for underserved and at risk populations where transition of care between multiple providers is not uncommon . This study demonstrated a positive impact of EHR adoption on immunization reporting to a citywide immunization registry. Benefits such as these are among the reasons that immunization reporting is included in Stage I requirements for meaningful use of EHRs . NYC CIR has recently implemented bi-directional HL7 messaging standards that will allow providers to attest to this meaningful use criterion. The enhanced interoperability between CIR and EHR systems that results will provide greater consistency in data access and exchange, and increase the availability of health information for other public health functions to assure community health (e.g. assessment of vaccine coverage during emergencies, spikes in communicable disease incidence, or outbreaks of seasonal illness) .
This study demonstrated efficiencies in automated reporting via an EHR to an immunization registry. Improvements included reduction in reporting lag time and an increase in automated reporting versus other methods. The ability of EHRs to improve reporting of health data to registries has implications for achieving public health goals for many at risk populations.
The authors declare no conflicts of interest in the research.
The study was approved by the Columbia University Institutional Review Board.
This study was conducted as part of HITEC (Health Information Technology Evaluation Collaborative), an academic consortium designated by the New York State as the evaluation entity for health IT projects funded under the HEAL-NY Law, a capital grant program to transform healthcare in New York State through technology and innovation. HITEC is a formal collaborative of researchers at Weill Cornell Medical College, Columbia University, the University of Rochester, and the State University of New York at Albany. This study was supported by the New York State Department of Health (NYS contract number C023699). Andrew B. Phillips received support from an institutional training grant to Columbia University from the National Institute of Nursing Research (T32NR007969).