Search tips
Search criteria 


Logo of appciLink to Publisher's site
Appl Clin Inform. 2013; 4(2): 267–275.
Published online 2013 June 12. doi:  10.4338/ACI-2013-02-CR-0009
PMCID: PMC3716417

Effects of Automated Immunization Registry Reporting via an Electronic Health Record Deployed in Community Practice Settings



Among the expected benefits of electronic health records (EHRs) is increased reporting of public health information, such as immunization status. State and local immunization registries aid control of vaccine-preventable diseases and help offset fragmentation in healthcare, but reporting is often slow and incomplete. The Primary Care Information Project (PCIP), an initiative of the NYC Department of Health and Mental Hygiene, has implemented EHRs with immunization reporting capability in community settings.

Objective and Methods

To evaluate the effect of automated reporting via an EHR on use and efficiency of reporting to the NY Citywide Immunization Registry, we conducted a secondary analysis of 1.7 million de-identified records submitted between January 2007 and June 2011 by 217 primary care practices enrolled in PCIP, pre and post launch of automated reporting via an EHR. We examined differences in records submitted per day, lag time, and documentation of eligibility for subsidized vaccines.


Mean submissions per day did not change. Automated submissions of new and historical records increased by 18% and 98% respectively. Submissions within 14 days increased from 84% to 87%, and within 2 days increased from 60% to 77%. Median lag time decreased from 13 to 10 days. Documentation of eligibility decreased. Results are significant at p<0.001.


Significant improvements in registry use and efficiency of reporting were found after launch of automated reporting via an EHR. A decrease in eligibility documentation was attributed to EHR workflow. The limitations to comprehensive evaluation found in these data, which were extracted from a registry initiated prior to widespread EHR implementation suggests that reliable evaluation of immunization reporting via the EHR may require modifications to legacy registry databases.

Keywords: Public health, electronic health records, immunization, health information exchange

1. Introduction

Immunization registries are confidential, population-based, computerized databases that record all immunization doses administered by participating providers to persons residing within a given geopolitical area [1, 2]. State and local registries offset fragmentation in healthcare by consolidating scattered records, by reducing unnecessary vaccination among patients who have lost records or visit multiple providers, and by tracking vaccine formulations, supply, and schedules [3]. In 2010, the Task Force on Community Preventive Services recommended registries to increase vaccination coverage based on strong evidence of their effectiveness for managing immunizations [4]. Registry data also are used to evaluate vaccine programs and population coverage [5-10]. However, accuracy, timeliness and completeness of data submitted are highly variable [9, 11-22], reducing the impact of these registries. Electronic health records (EHRs) provide a potential solution when immunizations documented at the point of care are reported automatically, directly to the registry [5, 23].

In 1997 the Citywide Immunization Registry (CIR) was established as a centralized repository of immunization records for people vaccinated in NYC. Over 2,000 medical practices and other venues are mandated to report immunizations administered to children 19 and older who reside in NYC. [19, 24]. There are three ways to submit immunization records to CIR:

  • 1.
    manual entry in CIR’s online registry;
  • 2.
    electronic transfer as a batch file, either automatically from an EHR or using data from a practice billing system; and
  • 3.
    direct real-time HL7 messaging, which currently is being implemented and not examined here [25].

Starting in 2007 EHRs equipped with an automated immunization reporting module were implemented in community practice settings by the Primary Care Information Project (PCIP). This initiative of the NYC Department of Health and Mental Hygiene subsidized adoption of EHRs, targeting practices serving communities at risk for health disparities determined by Medicaid visits per zip code [26]. Within the EHR a customized application, or module, automates the process of reporting immunization records via an Internet connection to CIR established when a practice first deploys the module. Each evening the module automatically aggregates the day’s records of immunizations administered and uploads a batch file directly to the CIR Web File Registry (WFR) [27]. This nightly process extracts immunization data directly from the native EHR database with no additional workflow for staff. The specifications for the module were developed collaboratively by PCIP and CIR and implemented by the vendor eClinicalWorks (eCW). At the time of the study immunization reports used a legacy reporting format, and CIR was planning transition to the national HL7 standard. To date, there are approximately 300 practices reporting to CIR via an automated immunization module.

We conducted a secondary analysis of immunization records submitted to CIR by a cohort of these practices to evaluate the effect of automated EHR-based reporting on use of the registry and efficiency of reporting. To our knowledge there are no previous studies of registry use and efficiency related to EHR-based reporting.

2. Methods

Practices were eligible if they were enrolled in PCIP, had implemented the immunization module in their EHR, and were certified by CIR to use the module to submit immunization records. In general, a practice is certified after approximately 30 days of rigorous system testing by CIR staff.

We obtained a de-identified, limited data set of vaccination records submitted to CIR by practices that met the study criteria. The data covered children ages 19 and under and represented all immunizations, including influenza, submitted to the CIR between January 1, 2007 and May 31, 2011. Records submitted in the 1.25 years before the CIR certification date for each practice were considered pre-implementation records. Records submitted in the 1.25 years after were considered post-implementation records. This time period was selected to reflect a full year of reporting before and after certification, including additional time (0.25 years) for transition pre/post. In all cases daily submission counts above the 99th percentile were removed to minimize skew in the distribution. MySQL was used to extract relevant data sets for analysis. SAS was used for statistical tests and to generate regression plots. The study was approved by Columbia University Institutional Review Board.

To measure effect on use of the registry we tested pre and post implementation differences in the mean number of records submitted per day via all sources, via the online registry, and via automatic file transfer. In the sub set of records submitted via automatic file transfer we examined new records (vaccines submitted ≤ 1 year of delivery) and historical records (submitted >1 year post delivery, or flagged as historical by the provider). These data were normally distributed and the student’s t-test was performed.

To evaluate the efficiency of reporting we excluded historical records from the analysis to eliminate the effect of large batches of backlogged records submitted by individual practices shortly after EHR immunization module deployment. We tested pre and post implementation differences in the median percentage of records from all sources uploaded within the 14-day requirement and within 2 days. Due to the nightly record upload a 2-day window represents immediate reporting via the module and reflects an increased use of the module. The 14-day reporting period is the legal reporting requirement and represents a broader measure of reporting efficiency by study practices. We tested pre and post implementation lag time, defined as the difference in days between the vaccine administration date and the date it was reported to the CIR. To approximate record completeness pre and post implementation we determined the median percentage of records submitted from all sources which had a non-null value in the field indicating eligibility for the Vaccine for Children program (VFC), a federal program that subsidizes vaccines. CIR tracks vaccines administered through this program. These data were asymmetrically distributed and the Wilcoxon rank sum test was performed. We also performed segmented regressions to allow visual inspection of pre/post data plots and to test for significant differences in slope.

3. Results

There were 217 practices that met the study criteria. Of these 82% (n = 178) had submitted records to CIR before implementation of automated reporting via the EHR. About 18% (n = 39) began submitting to CIR after implementation, despite the NYC reporting requirement. The data obtained consisted of 1.7 million immunization records representing 220,000 unique individuals. It should be noted that the population of children was stable during the study period, as shown in ► Table 1 [28].

Table 1
US census population estimates of children in NYC during the study period.

Although the mean number of records submitted daily to CIR from all sources remained constant, the pattern of submissions changed significantly. The results are shown in ► Table 2.

Table 2
Immunization records submitted to NYC Immunization Registry before and after by a set of primary care practices that were certified to submit immunization records automatically from their EHR. N = 178 practices reporting before, 217 practices reporting ...

Submissions via the online registry decreased from a mean of 1061 to 369 per day. Submissions via automated file transfer increased from mean of 1451 to 2051 per day. For new records automated file transfers increased from a daily mean of 917 to 1085 and for historical records from 497 to 985. Median lag time before implementation was 13.3 days [IQR = 10.7, 17.2], and after was 9.8 days [IQR = 6.4, 15.9]. Median lag time for manual submissions before implementation was 32.7 days [IQR = 25.9, 39.9], and after was 80.3 days [IQR = 61.6, 105.6]. The percentage of records submitted to CIR within the 14-day legal limit improved from 84% to 87%. The percentage of records submitted within 2 days increased from 60% to 77%. The proportion of records with the Vaccine for Children field completed decreased significantly after implementation, from 70% to 62%. These differences are significant at p<0.001.

A segmented regression plot in ► Figure 1 shows the percent of immunization records reported within 2 days of vaccine administration. The certification dates of each practice were aligned and outliers above the 99th percentile were removed. There is a visually notable decrease in dispersion of submissions after automated reporting with a significant pre/post difference in slope of the regression line (p<0.001).

Fig. 1
A segmented regression plot showing the percent of immunization records reported to the NYC Immunization Registry within 2 days of vaccine administration by 217 primary care practices. The certification dates for each practice were aligned. Outliers above ...

4. Discussion

The combined evidence suggests that automated reporting via the EHR has facilitated registry use for these 217 practices. The overall mean number of records submitted per day remained stable, suggesting already high compliance by providers with reporting mandates as well as suggesting a stable number of immunizations during the study. Automatic submissions (both new and historical) did increase in proportion to all submissions, and records from 39 previously non-compliant practices were captured. The increases in the proportion of submissions within 2 and within 14 days may underestimate the true efficiencies achieved since 18% of practices had not previously reported. The decreases in 2 and 14 day lag time are corroborated by the decreased data dispersion visualized post certification.

We did not assess how automated reporting affected workflow, and cannot speak to whether the gains in efficiency for CIR came at the expense of any clinician time required to use the reporting module. We might argue the greater lag in manual reporting that we found post implementation suggests a preference for automated reporting, but it could also suggest over reliance on automated reporting, which would represent an unintended consequence. As always, further research is needed to fully comprehend the effects of this, or any, implementation.

The decline in completeness of the VFC eligibility field may be due to multiple factors. Records submitted via the EHR did not contain a field for Medicaid numbers, a criterion from which eligibility may be determined. Further, a provider could document VFC eligibility in more than one field, but automated file transfer captured only one of those fields, suggesting that the EHR workflow may present difficulties for recording eligibility. CIR staff is evaluating these issues.

The study was greatly limited by the registry database structure: records submitted electronically via a non-EHR practice management system versus the EHR immunization module could not be dis-aggregated. This fact limited our analysis to pre- and post-certification of electronic reporting aggregated at the practice level. We were unable to discern if practices may have used non-EHR systems after certification to submit records electronically. We believe that situation is unlikely for two reasons: reporting via the EHR was automated to deliver records in a more timely way, and our findings clearly documented such a change. The data structure also limited our ability to ascertain a rate of immunization based on the patient population of the study practices. Significant changes in the underlying patient population could impact the interpretation of the study results. We don’t believe this to be the case as the the overall patient population was stable during the study period (► Table 1) and no significant changes in the mean immunizations reporting pre and post implementation were found (► Table 2). The structural limitation presents a cautionary lesson for those considering analysis of data from legacy immunization registries to evaluate reporting via the EHR. Perhaps our strongest finding is to illustrate that evaluation of the impact of automated immunization reporting will require the collection and reporting of additional data elements by EHR systems and registries to improve the accuracy and reliability of future studies.

5. Conclusion

Immunization recommendations in the US currently target 17 vaccine-preventable diseases across the lifespan [29]. As EHRs proliferate, automated reporting of immunizations can help public health officials and individual providers better determine who in the patient population has been adequately immunized, reduce provider paperwork and staff time, provide easy access and reliable immunization histories, and reinforce the concept of the medical home [30]. Improved efficiency of immunization registry reporting via an EHR is of increased importance for underserved and at risk populations where transition of care between multiple providers is not uncommon [32]. This study demonstrated a positive impact of EHR adoption on immunization reporting to a citywide immunization registry. Benefits such as these are among the reasons that immunization reporting is included in Stage I requirements for meaningful use of EHRs [31]. NYC CIR has recently implemented bi-directional HL7 messaging standards that will allow providers to attest to this meaningful use criterion. The enhanced interoperability between CIR and EHR systems that results will provide greater consistency in data access and exchange, and increase the availability of health information for other public health functions to assure community health (e.g. assessment of vaccine coverage during emergencies, spikes in communicable disease incidence, or outbreaks of seasonal illness) [5].

Clinical Relevance Statement

This study demonstrated efficiencies in automated reporting via an EHR to an immunization registry. Improvements included reduction in reporting lag time and an increase in automated reporting versus other methods. The ability of EHRs to improve reporting of health data to registries has implications for achieving public health goals for many at risk populations.

Conflicts of Interest

The authors declare no conflicts of interest in the research.

Human Subject Research Approval

The study was approved by the Columbia University Institutional Review Board.


This study was conducted as part of HITEC (Health Information Technology Evaluation Collaborative), an academic consortium designated by the New York State as the evaluation entity for health IT projects funded under the HEAL-NY Law, a capital grant program to transform healthcare in New York State through technology and innovation. HITEC is a formal collaborative of researchers at Weill Cornell Medical College, Columbia University, the University of Rochester, and the State University of New York at Albany. This study was supported by the New York State Department of Health (NYS contract number C023699). Andrew B. Phillips received support from an institutional training grant to Columbia University from the National Institute of Nursing Research (T32NR007969).


1. Papadouka V, Schaeffer P, Metroka A, Borthwick A, Tehranifar P, Leighton J. Integrating the New York Citywide Immunization Registry and the Childhood Blood Lead Registry. Journal of Public Health Management and Practice 2004; 10: S72 - S80 [PubMed]
2. Boom J, Dragsbaek A, Nelson C. The Success of an Immunization Information System in the Wake of Hurricane Katrina. Pediatrics 2007; 119(6): 1213 - 1217 [PubMed]
3. Freeman VA, DeFriese GH. The challenge and potential of immunization registries. Annual Review of Public Health 2003; 24: 227-246 [PubMed]
4. Committee on Practice of Ambulatory Medicine Immunization Information Systems. Pediatrics 2006; 118(3): 1293-1295 doi: 10.1542/peds.2006-1723 [PubMed]
5. Centers for Disease Control and Prevention Progress in Immunization Information Systems - United States, 2009. Morbidity and Mortality Weekly Report 2011; 60(1): 10-12 [PubMed]
6. Miller PL, Frawley SJ, Brandt C, Sayward FG. Tools for Immunization Guideline Knowledge Maintenance: II. Automated Web-Based Generation of User-Customized Test Cases. Computers and Biomedical Research 1998; 31(3): 190-208 doi: 10.1006/cbmr.1998.1471 [PubMed]
7. Kolasa MS, Chilkatowsky AP, Stevenson JM, al e. Do Laws Bring Children in Child Care Centers Up to Date for Immunizations? Ambulatory Pediatrics 2003; 3(3): 154-157 [PubMed]
8. LeBaron CW, Chaney M, Baughman AL, Dini EF, Maes E, Dietz V, et al. Impact of measurement and feedback on vaccination coverage in public clinics, 1988-1994. Journal of the American Medical Association 1997; 277(8): 631-635 [PubMed]
9. Khare M, Piccinino L, Barker LE, Linkins RW. Assessment of Immunization Registry Databases as supplemental sources of data to improve ascertainment of vaccination coverage estimates in the national immunization survey. Archives of Pediatric Adolescent Medicine 2006; 160(8): 838-842 [PubMed]
10. Salmon D, Smith P, Navar A, Pan W, Omer S, Singleton J, et al. Measuring immunization coverage among preschool children: past, present, and future opportunities. Epidemiology Review 2006; 28: 27-40 Epub 2006 Jun 1 [PubMed]
11. Stecher DS, Adelman R, Brinkman T, Bulloch B. Accuracy of a state immunization registry in the pediatric emergency department. Pediatric Emergency Care 2008; 24(2): 71-74 [PubMed]
12. Kwong J, Manuel D. Using OHIP physician billing claims to ascertain individual influenza vaccination status. Vaccine 2007; 25(7): 1270-1274 [PubMed]
13. Wilton R, Pennisi AJ. Evaluating the Accuracy of Transcribed Computer-Stored Immunization Data. Pediatrics 1994; 94(6): 902-906 [PubMed]
14. Ronveaux O, Arrieta F, Curto S, Laurani H, Danovaro-Holliday MC. Assessment of the quality of immunization data produced by the national individual registration system in Uruguay, 2006. Rev Panam Salud Publica 2009; 2: 153-160 [PubMed]
15. Pringle M, Ward P, Chilvers C. Assessment of the completeness and accuracy of computer medical records in four practices committed to recording data on computer. British Journal of General Practice 1995; 45(5): 537-541 [PMC free article] [PubMed]
16. Stecher DS, Adelman R, Brinkman T, Bulloch B. Accuracy of a state immunization registry in the pediatric emergency department. Pediatric Emergency Care 2008; 24(2): 71-74 [PubMed]
17. Davidson AJ, Melinkovich P, Beatty BL. Immunization Registry Accuracy – Improvement with progressive clinical application. American Journal of Preventive Medicine 2003; 24(3): 276-280 [PubMed]
18. Greene S, Ping Shi M, Dutta-Linn M, al e. Accuracy of Data on Influenza Vaccination Status at Four Vaccine Safety Datalink Sites. J Am of Preventive Medicine 2009; 37(6): 552-555 [PubMed]
19. Sy LS, Liu IL, Solano Z, Cheetham TC, Lugg MM, Greene SK, et al. Accuracy of influenza vaccination status in a computer-based immunization tracking system of a managed care organization. Vaccine 2010; 28(32): 5254-5259 Epub 2010 June 8 [PubMed]
20. Irving S, Donahue J, Shay D, Ellis-Coyle T, Belongia E. Evaluation of self-reported and registry-based influenza vaccination status in a Wisconsin cohort. Vaccine 2009; 27(47): 6546-6549 [PubMed]
21. Irigoyen M, Findley SE, Chen S, Vaughan R, Sternfels P, Caesar A, et al. Early Continuity of Care and Immunization Coverage. Ambulatory Pediatrics 2004;4(3): 199-203 [PubMed]
22. Kempe A, Steiner J, Refrew B, Lowery E, Haas K, Berman S. How Much Does a Regional Immunization Registry Increase Documented Immunization Rates at Primary Care Sites in Rural Colorado? Ambulatory Pediatrics 2001; 1(4): 213-216 [PubMed]
23. Kukafka R, Ancker J, Chan C, Chelico J, Khan S, Mortoti S, et al. Redesigning electronic health systems to support public health. Journal of Biomedical Informatics 2007; 40(4): 398 - 409 [PubMed]
24. New York City Health code section 11.04 and 11.07(d) [cited 2012 August 11] Available from:
25. Citywide Immunization Registry Web File Repository Guide 2010. Available from:
26. New York City Department of Health and Mental Hygiene Primary Care Information Project [cited 2012 August20]. Available from:
27. The City of New York. gov/html/doh/html/living/cir-how-to-report.shtml (2013, 1/30/2013). CIR: How to Report Retrieved 3/19/2013, from.
28. Contract # C023699. 2007-2010.
29. Centers for Disease Control and Prevention Vaccine and Immunizations: Immunization Schedules 2012. [cited 2012 August 11] Available from:
30. Centers for Disease Control and Prevention Vaccine and Immunizations: IIS Frequently Asked Questions 2012 [cited 2012 May 11]. Available from:
31. Centers for Medicare and Medicaid Services Incentive EHR Program: Department of Health and Human Services; 2011. [updated September 30, 2011; cited 2011 October 20] Available from:
32. Silow-Carroll S., Alteras T., Stepnick L. Patient-Centered Care for Underserved Populations: Definition and Best Practices: W.K. Kellogg Foundation; 2006. [cited 2012 May 22] Available from: http://www.esre

Articles from Applied Clinical Informatics are provided here courtesy of Schattauer Publishers