Search tips
Search criteria 


Logo of jamiaAlertsAuthor InstructionsSubmitAboutJAMIA - The Journal of the American Medical Informatics Association
J Am Med Inform Assoc. 1999 Mar-Apr; 6(2): 115–121.

Measuring the Effects of Reminders for Outpatient Influenza Immunizations at the Point of Clinical Opportunity


Objective: To evaluate the influence of computer-based reminders about influenza vaccination on the behavior of individual clinicians at each clinical opportunity.

Design: The authors conducted a prospective study of clinicians' influenza vaccination behavior over four years. Approximately one half of the clinicians in an internal medicine clinic used a computer-based patient record system (CPR users) that generated computer-based reminders. The other clinicians used traditional paper records (PR users).

Measurements: Each nonacute visit by a patient eligible for an influenza vaccination was considered an opportunity for intervention. Patients who had contraindications for vaccination were excluded. Compliance with the guideline was defined as documentation that a clinician ordered the vaccine, counseled the patient about the vaccine, offered the vaccine to a patient who declined it, or verified that the patient had received the vaccine elsewhere. The authors calculated the proportion of opportunities on which each clinician documented action in the CPR and PR user groups.

Results: The CPR and PR user groups had different baseline compliance rates (40.1 and 27.9 per cent, respectively; P < 0.05). Both rates remained stable during a two-year baseline period (P = 0.34 and P = 0.47, respectively). The compliance rates in the CPR user group increased 78 per cent from baseline (P < 0.001), whereas the rates for the PR user group did not change significantly (P = 0.18).

Conclusions: Clinicians who used a CPR with reminders had higher rates of documentation of compliance with influenza-vaccination guidelines than did those who used a paper record. Measurements of individual clinician behavior at the point of each clinical opportunity can provide precise evaluation of interventions that are designed to improve compliance with guidelines.

Attention to preventive health services is increasing as evidence for their efficacy accumulates.1 One of the clearest guidelines that reflects widespread consensus among clinicians is the recommendation to administer influenza vaccines to adults 65 years of age or older.2,3,4

Yet vaccination rates range between 45 and 58 per cent.5 Although health care providers have good in-Donald6 wrote in 1976 the “nonperfectibility of man” has prevented health care professionals from acting reliably on their own intentions.

Providers often inadvertently overlook effective preventive therapy for millions of people—an error of omission that contributes to tens of thousands of influenza-related deaths.7 A cohort study that measured adjusted outcomes of immunized compared with nonimmunized patients over a three-year period demonstrated that immunized patients have approximately half the deaths, half the hospitalization rates, and half the hospital costs of nonimmunized patients.5 Other studies showed similar efficacy rates.8 In a Canadian study, 80 per cent of patients 65 years of age or older who were hospitalized for influenza-associated respiratory conditions had had at least one visit with a health care provider prior to the admission and thus in many cases had presented the health care system with an opportunity to administer the influenza vaccine.9

Managed care health plans often require provider organizations to report how well their practitioners comply with clinical guidelines.10 Methods for reporting and auditing guideline compliance rates, however, are not clearly or consistently defined.

For most public health studies of influenza vaccination rates, either administrative billing data5 or consumer questionnaires11,12 are used to determine the rates of immunization. Although the denominators used in different methods of calculating the compliance rates vary, the numerator represents the number of consumers or patients who have received (or believe they have received) an influenza vaccine during the past influenza season. This method of calculating vaccination rates produces a valid rate for a defined population, but it does not permit investigators to study the effectiveness of an intervention designed to improve individual clinicians' compliance rates. For example, offering immunizations at churches, senior centers, or immunization clinics may increase vaccination rates even though providers may take no specific action. Increased media attention to influenza deaths during a given year may motivate patients to seek vaccinations from their providers. A more precise measure of clinicians' actions at each clinical opportunity would help researchers understand what tools help clinicians comply with their own intentions to administer preventive health services.

Several interventions have been developed to address the problem of underutilization of influenza vaccination in adults and children. Placing reminder notices on the front of the paper medical record is labor intensive and is subject to the same “nonperfectibility” problem.13 Similarly, mailing reminder postcards or letters has not produced consistently good results.14,15 Because health care providers can favorably influence patients' beliefs about and acceptance of influenza vaccination, reminding clinicians when they are seeing an eligible patient has additional benefits.16,17

Computer-based reminder systems have been shown to increase clinicians' rates of compliance with guidelines, especially in preventive health maintenance interventions.18,19,20,21,22,23,24 Although the methods used to measure rates of compliance vary across the studies, interventions using computer-based reminders generally use the entering of an influenza vaccination order as the satisfying action for the guideline. When we initially distributed individual compliance reports to our clinicians, using the metric by which only vaccination orders are equated with guideline compliance, they questioned the validity of the report. They argued that there are patient conditions and preferences that are not under the control of the clinician (e.g., patient declines the vaccine, patient received the vaccine elsewhere, patient is allergic to or has other contraindications for the vaccine). One of the reasons that these extenuating patient conditions are not typically considered in compliance measurements is that they are difficult to ascertain through review of the paper medical record. To overcome the limitations that paper-based records impose on practical measurement of influenza vaccination guideline compliance, we used a computer-based patient record system to capture these situations in a coded format and incorporated these considerations in our measure of compliance with influenza vaccination guidelines.


We conducted this study in a large internal medicine clinic at an academic center. Approximately one half of the clinicians had been using a computer-based patient record system (EpicCare, developed by Epic Systems, Madison, Wisconsin) as their primary record-keeping tool for almost two years (CPR users). The other clinicians in the clinic used traditional paper records (PR users). We defined the influenza season as October 1 through January 31 and referred to the season by the year in which it ended (in January). The CPR system was installed in July 1996, prior to the 1997 influenza season. Rule-based clinical reminders were developed and implemented in September 1997, just before the 1998 influenza season. The reminders appear when the clinician opens the chart of a patient who is eligible for a recommended intervention (Figures [triangle] and [triangle]).

Figure 1
Display of clinical reminders. When a patient's chart is opened, the system presents relevant alerts on the patient. Rules can be written to consider the patient's problems, medications, allergies, procedures, health maintenance, and laboratory test ...
Figure 2
Influenza vaccination order set. If the provider agrees with the recommended action, clicking “yes” on the screen shown in [triangle] brings up a dynamically configured set of actions with defaults preselected (indicated by check marks). ...

Before implementing the clinical reminder system, we distributed a baseline survey of clinicians' attitudes about recommendations regarding preventive health services to all 34 clinicians in the clinic; 23 (68 per cent) returned it. One question was whether they agreed with the recommendation to offer influenza vaccination to adults 65 years of age or older who do not have contraindications to the vaccine.

We used scheduling data to identify all patients 65 years of age and older who had one or more nonacute clinic visits during the influenza seasons of each year from 1995 through 1998. We divided the pool of patient visits into those who saw a CPR user and those who saw a PR user. Patients who had received vaccinations in September (i.e., prior to the defined influenza season) were excluded, as were patients who were allergic to a vaccine component or who had another contraindication. There were no statistically significant differences in the distribution of chronic medical conditions (congestive heart failure, pulmonary diseases, diabetes, coronary artery disease, or stroke) among the patients of the two provider groups. Data were abstracted from the medical records and entered into Paradox version 4.5; they were then imported to SPSS version 7.0 for statistical analysis.

We used the PR as the source document during the prestudy period for both groups; we used the CPR as the source document for the CPR user group and the PR as the source for the PR user group during the study period.

We assessed what actions clinicians took when they were presented with an opportunity to follow the influenza guideline, by measuring the rate at which each clinician offered eligible patients an influenza vaccination. Each visit by a patient eligible for an influenza vaccination was considered an opportunity for the clinician to comply with the guideline by performing one of the following four actions: ordering the vaccine; documenting that counseling was performed; documenting that the vaccine was offered, but the patient declined; or documenting that the patient had already received the vaccine elsewhere. After a patient received or declined the vaccine, he or she was no longer considered an eligible patient for the remainder of that year's influenza season. If only counseling was offered during a visit or if a time-limited contraindication was present, the patient remained eligible for vaccination.


We attempted to abstract 100 per cent of the records of all eligible patients for each study year. The numbers of patient visits in each study group are summarized in [triangle]. Despite multiple attempts, however, we could not locate the records or find documentation for a portion of the patient visits. [triangle] summarizes the number of patient records or encounter documents that we were not able to retrieve.

Table 1
Eligible Patients Seen by Clinicians in Computer-based Patient Record (CPR) Group and Those in Paper-based Records (PR) Group
Table 2
Missing Encounter Documents in Computer-based Patient Record (CPR) Group and Paper-based Records (PR) Group

Of the 23 participants (13 CPR users and 10 PR users) who returned the preventive services survey, 95 per cent either agreed or strongly agreed with the recommendation that adults 65 years of age or older who had no contraindications should be offered an influenza vaccination during the influenza season. The influenza guideline compliance rates are summarized in [triangle] and in [triangle].

Table 3
Rates of Compliance with Influenza Vaccination Guidelines in Computer-based Patient Record (CPR) Group and Paper-based Records (PR) Group
Figure 3
Rates of compliance with influenza vaccination guidelines among clinicians using a computer-based patient record system (solid black squares indicating intervention group) and those using traditional paper-based records (open triangles indicating control ...

There were no statistically significant changes from the first (1995) to the second (1996) year of baseline measurements for either the CPR users or the PR users (P = 0.36 and P = 0.47, respectively). Both groups increased their compliance rates in 1997 compared with 1996 (59 per cent, P < 0.001, and 30 per cent, P < 0.02, respectively). In 1998, the CPR user group's compliance rate increased again (P < 0.02), for a total increase of 78 per cent (P < 0.001 by the Kruskal-Wallis test) over the 1996 rate prior to the introduction of a CPR. The PR user group's compliance rate dropped by 17 per cent (P < 0.03) in 1998 compared with 1997. The final PR user group compliance rate in 1998 was similar to the baseline in 1996 (P = 0.18).


To evaluate an intervention designed to encourage administration of influenza vaccines for eligible patients, we developed a metric to assess clinicians' behavior at each opportunity to offer the vaccine. We refined existing methods of measuring responses to computer-based reminders by capturing patient conditions and preferences that affect appropriate clinician behavior in following the influenza-vaccination guideline. In contrast to those based on population data alone, this measure allowed us to quantify more precisely changes in the behavior of individual clinicians that were attributable to computer-based reminders. We believe this metric for assessing clinicians' behaviors at each clinical opportunity can be applied to other clinical guidelines.

Because of the personal time required to learn to use a CPR, the CPR user group was made up of clinicians who volunteered to use a CPR as part of a pilot project. Consequently, our results may not generalize to the broad population of all clinicians. Although the baseline compliance rates of the CPR user group were higher than those of the PR user group, each group's rates were stable during the two years prior to the implementation of computer-based reminders. The stability of the rates gave us confidence that, without any other external changes, the clinician group's compliance rates would remain approximately the same from year to year.

The statistically significant increases in compliance rates for both groups that occurred between 1996 and 1997 are interesting. During the 1997 influenza season, for reasons unrelated to the study, clinical memoranda informed all clinicians that the major health plans had requested reports on the practice's compliance with preventive health services, such as administration of the influenza vaccine. The practice had also set up a table in the building lobby to promote the clinic and remind people about influenza vaccinations. The public media ran stories about how severe influenza infections were expected to be that season. In contrast, because of an unseasonably warm winter in 1998, media attention to influenza vaccination waned. Although we do not have a definitive explanation for why the PR user group's compliance rate increased in 1997, it is notable that the level of increase was only half that of the CPR user group, and that whatever effect led to the increase in 1997 disappeared in 1998, while the CPR user group's rate continued to increase. We interpret the magnified positive increase among the CPR user group as a positive reflection of how use of a CPR can improve the degree to which clinicians act on their own intentions to follow guidelines. We interpret the continued rise in compliance rates, at a time when compliance by the PR user group returned to baseline rates, as evidence that computer-based reminders can further enhance the compliance rates.

Notice that this study measured documentation of actions related to influenza vaccination guidelines and does not report the actual percentage of patients who received the vaccine in either the CPR or the PR group. Only by directly observing or recording actual clinician behavior, neither of which we performed in this study, would we be able to determine whether CPR users and PR users interact differently with patients or whether that interaction affects clinical outcomes.

We argue that an increase in documentation is in and of itself an improvement in practice. The fact that the documented behaviors of both groups did change in 1997 indicates that both groups can change their documentation rates. Others have demonstrated that CPRs can improve documentation behavior as well.25 In addition, documentation is rising in importance both for quality reporting and for reimbursement compliance.

Conducting a study using a CPR system also had logistical advantages over paper chart reviews. For example, during medical record abstraction, we could not locate 20 to 31 per cent of the paper medical record documents. In contrast, there were no missing CPR documents. These benefits should extend to routine measurement of quality information, which is vastly easier with computer-based records.

As more clinical guidelines view each encounter as an opportunity to provide preventive health services or promote healthy behavior, providers will need effective and efficient methods to take advantage of each opportunity to enhance patient care. For example, health plans now request that clinicians not only document the smoking history of patients but also counsel patients who smoke about the health risks of smoking at each opportunity. The CPR is a valuable tool whose reminders can help clinicians act on these opportunities. Through the use of coded entries, it also automates the capture of precise, accurate information that helps quality personnel as well as researchers evaluate the effects of interventions designed to help improve compliance with guidelines, at every point of opportunity.


The authors thank their project team colleagues who implemented the computer-based patient record system in the clinic: Wendy Marquardt, Barbara Boggs, Cynthia Fellencer, Mary Jaworski, Nancy Kreider, Patricia Sandlund, and James Yarnold. They thank the members of the Information Services Division of Northwestern Memorial Hospital for their technical support of the computer-based patient record system. They also thank the clinicians at the General Internal Medicine Clinic at Northwestern Medical Faculty Foundation for their participation in the study. They are especially grateful to Gary J. Martin, MD, chief of the Division of General Internal Medicine, whose ever-present support made this project possible. They thank Lyn Dupré for her excellent editorial assistance.


This research was conducted and the paper written when Dr. Tang was a full-time employee of Northwestern Memorial Hospital. He is currently at the Palo Alto Medical Foundation and is a part-time employee of Epic Systems, the vendor of the CPR software.

This work was supported in part by High Performance Computing and Communications contract N01-LM-43509 from the National Library of Medicine.


1. Office of Disease Prevention and Health Promotion. Clinician's Handbook of Preventive Services. 2nd ed. U.S. Department of Health and Human Services, 1998.
2. American College of Physicians Task Force on Adult Immunization and Infectious Diseases. Influenza. In: Guide for Adult Immunization. 3rd ed. Philadelphia, Pa.: ACP, 1994: 91-4.
3. Canadian Task Force on the Periodic Health Examination. Prevention of influenza. In: The Canadian Guide to Clinical Preventive Health Care. Ottawa, Ontario, Canada: Minister of Supply Services, 1994.
4. U.S. Preventive Services Task Force. Adult immunization. In: Guide to Clinical Preventive Services. 2nd ed. Washington, D.C.: U.S. Department of Health and Human Services, 1996.
5. Nichol KL, Margolis KL, Wuorenma J, Von Sternberg T. The efficacy and cost effectiveness of vaccination against influenza among elderly persons living in the community. N Engl J Med. 1994;331:778-84. [PubMed]
6. McDonald CJ. Protocol-based computer reminders, the quality of care and the nonperfectability of man. N Engl J Med. 1976;295:1351-5. [PubMed]
7. Lui KJ, Kendal AP. Impact of influenza epidemics on mortality in the United States from October 1972 to May 1985. Am J Public Health. 1987;77:712-6. [PubMed]
8. Gross PA, Hermogenes AW, Sacks HS, Lau J, Levandowski RA. The efficacy of influenza vaccine in elderly persons: a meta-analysis and review of the literature. Ann Intern Med. 1995;123:518-27. [PubMed]
9. Fedson DS, Wajda A, Nicol JP, Roos LL. Disparity between influenza vaccination rates and risks for influenza-associated hospital discharge and death in Manitoba in 1982-1983. Ann Intern Med. 1992;116:550-5. [PubMed]
10. National Commission on Quality Assurance. Health Plan Employer Data and Information Set (HEDIS) 3.0. Annapolis Junction, Md.: NCQA, 1998.
11. Centers for Disease Control and Prevention. Pneumococcal and influenza vaccination levels among adults aged ≥65 years: United States, 1993. Atlanta, Ga.: CDC, 1996. [PubMed]
12. National Commission on Quality Assurance. Flu shots for older adults. Health Plan Employer Data and Information Set (HEDIS) 3.0. Annapolis Junction, Md.: NCQA, 1998:28-9.
13. Szilagyi PG, Rodewald LE, Humiston SG, et al. Reducing missed opportunities for immunizations: easier said than done. Arch Pediatr Adolesc Med. 1996;150:1193-200. [PubMed]
14. Buchner DM, Larson EB, Whilte RF. Influenza vaccination in community elderly: a controlled trial of postcard reminders. J Am Geriatr Soc. 1987;35:755-60. [PubMed]
15. Moran WP, Nelson K, Wofford JL, Velez R. Computer-generated mailed reminders for influenza immunization: a clinical trial. J Gen Intern Med. 1992;7:535-7. [PubMed]
16. Honkanen PO, Keistinen T, Kivela SL. Factors associated with influenza vaccination coverage among the elderly: role of health care personnel. Public Health. 1996;110:163-8. [PubMed]
17. Nichol KL, Lofgren RP, Gapinski J. Influenza vaccination: knowledge, attitudes, and behavior among high-risk outpatients. Arch Intern Med. 1992;152:106-10. [PubMed]
18. Sullivan F, Mitchell E. Has general practitioner computing made a difference to patient care? A systematic review of published reports. BMJ. 1995;311:848-52. [PMC free article] [PubMed]
19. Shea S, DuMouchel W, Bahamonde L. A meta-analysis of 16 randomized controlled trials to evaluate computer-based clinical reminder systems for preventive care in the ambulatory setting. J Am Med Inform Assoc. 1996;3:399-409. [PMC free article] [PubMed]
20. McDonald CJ, Hui SL, Smith DM, et al. Reminders to physicians from an introspective computer medical record: a two-year randomized trial. Ann Intern Med. 1984;100:130-8. [PubMed]
21. McDonald CJ, Hui SL, Tierney WM. Effects of computer reminders for influenza vaccination on morbidity during influenza epidemics. MD Comput. 1992;9:304-12. [PubMed]
22. McDonald CJ, Overhage JM, Tierney WM, Abernathy GR,j Dexter PR. The promise of computerized feedback systems for diabetes care. Ann Intern Med. 1996;124:170-4. [PubMed]
23. Overhage JM, Tierney WM, McDonald CJ. Computer reminders to implement preventive care guidelines for hospitalized patients. Arch Intern Med. 1996;156:1551-6. [PubMed]
24. Tierney WM, Overhage JM, Takesue BY, et al. Computerizing guidelines to improve care and patient outcomes: the example of heart failure. J Am Med Inform Assoc. 1995;2:316-22. [PMC free article] [PubMed]
25. Garrett LEJ, Hammond WE, Stead WW. The effects of computerized medical records on provider efficiency and quality of care. Methods Inf Med. 1986;25:151-7. [PubMed]

Articles from Journal of the American Medical Informatics Association : JAMIA are provided here courtesy of American Medical Informatics Association