During the 2011–2012 influenza immunization campaign, all immunizations administered at HRHD public clinics and all staff immunizations and declinations at RT were documented on scannable forms.
The time required to scan and verify forms at HRHD was significantly shorter than manual data entry, whereas at RT there was no difference (). Scannable Office has a batch scan setting that allows several forms to be scanned consecutively before each individual form is verified, leaving users free to engage in other tasks. Neither organization used this feature, but when we excluded the time required to scan each form and compared verification-only to manual data capture, the processing time for scanned forms was significantly faster than manually entered forms for both organizations.
Data capture: mean time (in seconds) per immunization record.
One RT staff member was responsible for scanning and verifying all forms, but at HRHD, multiple individuals performed these tasks. We observed two of these staff members, and when we analyzed time by user we found no difference in mean time for scanning and verification, but did detect a difference between users’ manual data entry time (mean difference: 3.9 s (95% CI: 1.0, 6.9)).
Record audits revealed high levels of agreement between paper forms and their respective electronic records for both scanned and manually entered data (). The individual data fields reported in were selected primarily based on clinical importance, although we limited our reporting of HRHD’s high-risk/priority tick-box fields to a subset whose likelihood of error we deemed to be equivalent (over 65 years, provider of essential services). The number of discordant pairs observed in some fields in the scanned group was slightly higher compared to those that were manually entered, although these differences were not statistically significant.
Agreement between paper forms and electronic records by data entry approach.
When we grouped all audited data elements into hand-print and tick-box fields, we found higher disagreement in handprint fields when scanned compared to when manually entered in both organizations, although these differences were not significant. We also observed non-significant increases in discordance in scanned tick-box fields for RT and in manually entered tick-box fields for HRHD.
We considered the possibility that some records containing multiple errors were responsible for a greater proportion of overall errors, but found that of the 494 records audited, six scanned and three manually entered forms each contained two errors. All remaining records contained one or no errors.
We assessed the usability of the scanning approach by examining the perceptions of users engaged in implementation, form completion, and scanning and verification roles. At HRHD, we interviewed one manager, two database personnel, four vaccine administrators, one staff member responsible for scanning/verification, and one individual who was involved in database design as well as scanning/verification, and administered questionnaires to two charge nurses, two clinic facilitators, and 198 clients. At RT, we conducted interviews with one manager and one staff member responsible for vaccine administration and scanning/verification.
The database personnel interviewed at HRHD were involved with form design and integrating the form’s field structure with the MSSQL database; one of these respondents was also involved with scanning. These individuals described the initial transfer of scanned data to Access as very straightforward, but indicated that creating an MSSQL database and appropriately mapping the scannable form’s field structure to this database required technical support from AutoData®. After initial difficulties had been overcome, however, database personnel were comfortable addressing any issues that arose.
Questionnaires were completed by 198 clients at HRHD clinics; the majority (88%) of respondents were 31 to 80 years of age, and 50% were female. Most clients reported that instructions were clear (81%), the experience of writing letters/numbers in individual boxes was the same or easier than other forms (88%), completing this type of form took the same amount of time or less than other types of forms (87%), and there were no parts of the form that they found confusing (84%). Chi-square tests did not reveal any statistically significant associations between responses to those questions and age or sex.
Some clients noted that these forms contained more space than others, and that the allocation of one box per character made the form easier to complete, while others commented that boxes were too small and staying inside the lines of each box was difficult. Notable recommendations included incorporating clearer instructions about completing tick-boxes with “X”s, allowing check marks to be used, increasing the character box size, and offering clipboards to improve ease of completion while waiting in line. While few clients approached clinic staff with questions about the scannable elements on the immunization consent forms, the observations that nurses and clinic facilitators made regarding areas of difficulty for clients corroborated these findings.
Nurses described varying form completion experiences; some indicated that the forms were easy to complete, including one who preferred the individual text boxes because they compelled her to write more neatly, while others found that the new format took some time to get used to, and said that it was sometimes hard to stay within the lines of each box.
All users who were involved with scanning and verifying forms found the procedures user-friendly, although some form processing difficulties were observed. In both organizations, the software was unable to recognize a small proportion of forms; sometimes this was the result of wrinkled or soiled pages, but in most cases it was not possible to determine why the error had occurred. When the first page of a form scanned correctly but the second page was unscannable, care had to be taken to ensure that the contents of the first page were not duplicated upon manual entry. Close to half of RT forms’ locator symbols had been skewed during printing, which necessitated manual entry of their contents (these were timed as manual entries, not scanned entries). Occasionally, multiple pages would be scanned at once which was problematic because it meant that the first page of one form and the second page of another were considered the same form; in these cases the entry would be cancelled, and the forms separated and rescanned.
The software’s initial interpretation of the content of the hand-printed fields often required correction at the point of verification. Some fields, including names and addresses, required more corrections than others during this phase, likely because of the length of the field, and in the case of address, because the field was not restricted to either alphabetic or numeric characters. Further, several barcodes on HRHD forms were not recognized and had to be manually entered. Commenting on the concentration that is required during verification, one individual who was involved with scanning highlighted that the quality of the data exported was “very reliant on the attention of the person who was authorizing it.”
When asked to compare their scanning and manual data entry experiences, users at RT described similar, positive perceptions of both approaches, while in HRHD scanning was perceived to be preferable, due to speed of data entry as well as usability.
At the conclusion of this pilot, RT and HRHD had each established a comprehensive electronic dataset of vaccinees, and users in both organizations acknowledged that the new approach to data collection afforded greater ease of access to many data elements. However, each organization differed in their attitudes about continuing to employ a similar approach in the future, based on perceived usability of more detailed data. HRHD was able to access required data (aggregate counts of age, sex, and high risk status) through traditional means (paper forms), and some program staff did not feel that the expenditure of resources was worthwhile for the influenza immunization program, especially since the vaccine must be administered every year. Further, because they provide a small proportion of all influenza vaccinations administered in the region, HRHD recognized that the coverage data they were able to assess were not representative of the entire population, and therefore limited in value. However, scanning as a mechanism for populating immunization registries was perceived as potentially valuable for other immunization programs that are administered exclusively by public health, because the resulting dataset would reflect the entire vaccinated population.
In contrast, as an institution requiring information about the influenza immunization status of each staff member, RT expressed a desire to continue to capture data electronically, whether through scanning or direct manual entry. The electronic availability of these data allowed management to assess coverage by department and level of patient contact – valuable for monitoring uptake throughout each campaign as well as emergency planning in the event of an outbreak – and also facilitated rapid data sharing with other personnel.
RT management acknowledged that the cost of the software ($4,670 USD) would likely be prohibitive for their small facility if it were used exclusively for immunization information, but that it would be possible to explore sharing data processing tasks with other associated institutions, and/or to use the program for other data collection needs.