Search tips
Search criteria 


Logo of jamiaAlertsAuthor InstructionsSubmitAboutJAMIA - The Journal of the American Medical Informatics Association
J Am Med Inform Assoc. 2008 Nov-Dec; 15(6): 776–779.
PMCID: PMC2585533

Use of Clinical Decision Support to Increase Influenza Vaccination: Multi-year Evolution of the System


Despite recognition that clinical decision support (CDS) can improve patient care, there has been poor penetration of this technology into healthcare settings. We used CDS to increase inpatient influenza vaccination during implementation of an electronic medical record, in which pharmacy and nursing transactions increasingly became electronic. Over three influenza seasons we evaluated standing orders, provider reminders, and pre-selected physician orders. A pre-intervention cross-sectional survey showed that most patients (95%) met criteria for vaccination. During our intervention, physicians were increasingly likely to accept pre-selected vaccination orders, Year 1 (47%), Year 2 (77%), Year 3 (83%); however vaccine administration by nurses was suboptimal. As electronic medical record functionality improved, patient receipt of vaccine increased dramatically, Year 1 [0/36; 0%], Year 2 [8/66; 12%], Year 3 [286/805; 36%]. Successful use of clinical decision support to increase inpatient influenza vaccination only occurred after initiation of CPOE for all medications and integration of an electronic medication administration record. Also, since most patients met criteria for influenza vaccination, complicated logic to identify high-risk patients was unnecessary.


Many improvements in patient-care derived from implementation of electronic medical records result from use of clinical decision support (CDS) systems triggered by computerized physician order entry (CPOE). 1–4 Despite the benefits realized from use of electronic medical records, CPOE, and CDS, penetration of these systems in U.S. healthcare settings remains low. 4–6 Increased adoption of CDS likely will follow demonstrations of successful implementations in institutions that have vendor-provided solutions.

Although influenza vaccination effectively prevents disease and reduces the risk of hospitalization, many high-risk people are not immunized. 7–9 As such, the Joint Commission on Accreditation of Healthcare Organizations measures influenza vaccination of eligible pneumonia inpatients as an indicator of good quality healthcare. For these reasons, and because the population served by Stroger Hospital (formerly Cook County Hospital) historically has been undervaccinated, 10 we designed a CDS-based intervention to increase influenza vaccination of hospital patients. Since components of our electronic medical record were introduced incrementally over three influenza seasons; we evaluated use of CDS during the maturation of our system. We report our challenges and successes during the implementation of our system.


Setting and Project Description

Stroger Hospital is a 464-bed public hospital. In 2001, we began installation of a new information system (Cerner Inc., Kansas City, MO.). Our information system gradually evolved as components were transferred from paper-based to electronic systems. We used the CDS system to improve influenza vaccination during the 20032004, 20052006, and 20062007 influenza seasons, i.e., Year 1, Year 2, and Year 3, respectively. Due to a nation-wide influenza vaccine shortage, there were no interventions during 20042005. We focused our intervention on internal medicine ward patients, who were admitted to one of three separate teams; admissions were consecutively assigned to a team, resulting in similar patient characteristics between teams. There were no education sessions for physicians during the three influenza seasons. We obtained approval from the IRB.

Year 1: 2003–2004 Influenza Season

During Year 1, we performed a cross-sectional survey to determine how often patients met high-priority category criteria for receipt of influenza vaccination; 11 we performed bedside interviews.

Computerized provider order entry (CPOE) was available for laboratory tests, radiographic studies, diets, electrocardiograms, and admission/discharge orders; there was no medication CPOE. Nursing documentation was paper-based, including medication administration. Electronic orders were printed at the nursing station. No standing orders policy existed for influenza vaccination ([triangle]).

Table 1
Table 1 Interventions Used to Increase Vaccination of Hospital Patients and Information System Maturation over Three Influenza Seasons

The cross-sectional survey showed that >95% of internal medicine patients met a high-priority criterion for vaccination; therefore, our CDS rule targeted all internal medicine inpatients. We tested separate strategies for each internal medicine team, as follows: 1) A pre-selected order “Administer Influenza Vaccine 0.5 ml IM” was presented to physicians in real time as a pop-up, triggered by their “Discharge Patient” order; 2) a pop-up reminder to order vaccine was presented to physicians upon entry of the “Discharge Patient” order; 3) no intervention, i.e., usual care.

Year 2: 2005–2006 Influenza Season

During Year 2, we augmented CDS rules with a written universally-applied standing-orders policy that enabled nurses to vaccinate inpatients even if there wasn't a patient-specific physician's order. We educated nurses about the policy and importance of influenza vaccination. Nurses viewed electronic orders through a patient-centered task list. Compared to Year 1, physicians entered more orders electronically, and increasingly documented patient evaluations in the electronic medical record. However, medication order entry still was paper-based ([triangle]).

We compared the following strategies concurrently (i.e., one strategy per team): 1) A pre-selected order “Administer Influenza Vaccine 0.5 ml IM” was presented to physicians, triggered by the “Discharge Patient” order; 2) an electronic reminder to follow the standing orders policy populated nurses' task lists upon patient admission; 3) usual care.

Year 3: 2006–2007 Influenza Season

During Year 3, medication CPOE was active and orders were routed electronically to the electronic medication administration record ([triangle]). Since despite physician orders, vaccine administration was uncommon during Year 2, in Year 3 we intended to test triggering the order at the time of patient admission. Unfortunately, we were unable to implement CDS using the “Admission to Bed” order; therefore, for Year 3, all results were for the “Discharge Patient” order—the same intervention as during Year 2.

Influenza Vaccination Assessment

During all three seasons, the pre-selected influenza vaccine order was enacted unless the physician de-selected the order. During Years 1 and 2, to determine patient vaccination receipt, we reviewed paper medication administration records and nursing notes on a random sample of patients. During Year 3, since vaccine administration was only documented electronically, we retrieved these data from the hospital information system for all patients; patient sampling was unnecessary.

To demonstrate changes in vaccination rates over time—during the transition from a hybrid paper-electronic system to an electronic system—we present the results from the intervention strategy that remained constant (i.e., pre-selected order triggered by the “Discharge Patient” order). We tested the trend in vaccination receipt by calculating the chi-square test for trend. All analyses were performed using Stata version 9.2 (StataCorp, College Station, TX).


Year 1

From the cross-sectional survey, approximately one-third of patients were vaccinated before hospitalization; reactions to eggs or previous influenza vaccination were rare and most patients met criteria that placed them in a high-priority category for vaccination ([triangle]). 11 The most common criterion met was age >49 years followed by diabetes, cardiac, pulmonary, or renal disease. For seven patients (7%) the sole criterion met could only be discerned through bedside interview; for example, living with a high-risk person or child aged less than two years.

Table 2
Table 2 Characteristics of Patients Evaluated during the Cross-sectional Survey, 2003–2004

During Year 1, of 114 patients sampled for chart review, none were vaccinated. Of 36 patients admitted to the pre-selected order team, 17 (47%) times the physician accepted the order; however, the vaccine never was administered, [triangle].

Figure 1
Influenza vaccination rates of hospital patients, by intervention and year.

Year 2

Of 204 patients sampled, 66 were admitted to the pre-selected order medical team. Compared to Year 1, physicians were much more likely to accept the pre-selected order (77% vs. 47%; p = 0.002) and patients were more likely to receive vaccine (8/66 [12%] vs. [0/36] 0%, p = 0.03), [triangle]. Despite having a standing orders policy, few patients admitted to the usual care or nursing reminder teams received influenza vaccine (1% and 6%, respectively).

Year 3

During Year 3—after implementation of the electronic medication administration record—among 805 patients discharged by the pre-selected order team, most physicians (n = 665; 83%) accepted the pre-selected order, and there was a dramatic increase in patient vaccination (n = 286; 36%), [triangle]. Of 665 patients who had a physician's order for vaccination, 43% were vaccinated. Since approximately 30% of patients were vaccinated before hospitalization, we estimate that 61% of eligible patients were vaccinated. Over the three influenza seasons, there was a significant increase in patient vaccination (Year 1 [0/36], Year 2 [8/66], Year 3 [286/805]; p < 0.001), [triangle].


Over time, physicians were increasingly likely to accept pre-selected vaccination orders; by Year 3 83% of orders were accepted. A bigger challenge was improving vaccine administration after physician order, which improved dramatically after the medication administration record was integrated into the electronic medical record. Also, since most internal medicine patients met criteria for vaccination, sophisticated rule-building was unnecessary. In fact, since not all patients who meet high-priority criteria for vaccination could be identified using clinical data, building rules to selectively trigger the CDS system would have resulted in missed opportunities to vaccinate high-risk patients.

Our findings illustrate the tenet that to successfully implement CDS it is essential to address workflow integration, healthcare worker-system interaction, local culture, and transition of most processes to the electronic system. 1,3,12–15 During Year 1, we attempted CDS in a predominantly paper-based system; for example, electronic orders were printed at the nurses station, which delayed notification of nurses about the order during a time critical process. Since our CDS rule was triggered by the “Discharge Patient” order, nurses were required to vaccinate patients in the relatively short time between the discharge order and the patient's departure. Although we considered other CDS triggers, each potential solution had challenges. For example, during Year 2 we tested automated electronic reminders to nurses, triggered by patient admission. Unfortunately, nurses rarely followed the policy. We considered using temporal triggers, e.g., orders presented by hospital day, but this was not an option with our CDS system. Finally, we attempted to use the admission order as a trigger for the pre-selected order, but could not resolve technical difficulties.

During Year 1, physicians who were exposed to a reminder, rather than the pre-selected order, did not place the order. Likely because in part, there was no medication CPOE, and physicians either had to write the order in the paper chart or search for the electronic order at a time when there was no CPOE for medications. During Year 2, increased vaccine administration by nurses likely resulted from increased functionality of the electronic medical record; for example, availability of an electronic task list for nurses. In Year 2, despite improved physician acceptance of the order and increased vaccine administration, coverage levels remained low.

Since nursing and physician reminders were unsuccessful during Years 1 and 2, in Year 3 we focused on using pre-selected physician orders. To facilitate nursing administration of vaccine during the discharge process, we intended to test presenting the pre-selected order to physicians upon both patient admission and discharge. However, despite successful use of the rule in the test environment, we were unable to reliably use the order triggered by patient admission. Specifically, influenza orders triggered by patient admission usually were not accepted by the system; therefore, the pharmacist could not verify the order electronically. Since Year 3 vaccination rates were dramatically improved, we believe that the low vaccine administration rates during Years 1 and 2 were primarily due to incomplete maturation of the electronic system, rather than logistical challenges posed by triggering the pre-selected order on patient discharge.

Our experiences illustrate how CDS implementations require attention to local workflow, in particular, transition of workflow from paper-based to electronic systems. This may partly explain the relatively low penetration of CDS in healthcare settings, especially since many institutions do not have the expertise to tailor less flexible vendor-provided systems to their needs. 4,16 Despite these challenges, we achieved significant and meaningful increases in vaccination coverage using CDS. By Year 3, over 50% of patients not vaccinated before hospitalization, were vaccinated during their hospital stay.

At our hospital, a written universally-applied standing orders policy was ineffective, even after augmenting the policy with electronic reminders to nurses. In addition to nurses' concerns about acting without an individual physician's order, during educational sessions many nurses expressed concerns about the influenza vaccine. After recognizing these substantial local barriers, during Year 3 we abandoned electronic nursing reminders—despite proven success at another institution. 17 For the following reasons, we presented the pre-selected order to all patients: incomplete electronic problem lists compromise the sensitivity of electronic inferences of chronic medical conditions, some criteria are not electronically captured (e.g., living with a high-risk person), we were hesitant to increase nurses' workload by creating an electronic form to screen patients, and influenza vaccination is safe and effective for all patients.

Our findings are limited in that we evaluated patients in a single, large, urban public hospital, nurses at other hospitals may be more likely to respond to electronic reminders and standing orders policies. 17 Also, we did not determine previous vaccination history through patient interview; however, a bedside interview may have influenced patients' desire for vaccination and biased our results.

We encountered several challenges during implementation of a CDS rule to increase influenza vaccination. These challenges included local cultural issues—nurses were reluctant to carry out standing orders—and technical issues, such as incomplete integration of our electronic medical record and lack of functionality of a vendor-provided system. Despite these challenges, we observed early and nearly complete physician acceptance of the CDS-generated order. After integration of the electronic medication administration record, there was a dramatic increase in nurses' administration of vaccine. Use of CDS can dramatically improve patient care, but success may be realized only after understanding the local workflow and culture, and near-complete transition from paper to electronic processes.


1. Gross PA, Bates DW. A pragmatic approach to implementing best practices for clinical decision support systems in computerized provider order entry systems J Am Med Inform Assoc 2007;14:25-28. [PMC free article] [PubMed]
2. Kuperman GJ, Bobb A, Payne TH, et al. Medication-related Clinical Decision Support in Computerized Provider Order Entry Systems: A Review J Am Med Inform Assoc 2007;14:29-40. [PMC free article] [PubMed]
3. Bates DW, Kuperman GJ, Wang S, et al. Ten Commandments for effective clinical decision support: making the practice of evidence-based medicine a reality J Am Med Inform Assoc 2003;10:523-530. [PMC free article] [PubMed]
4. Osheroff JA, Teich JM, Middleton B, Steen EB, Wright A, Detmer DE. A Roadmap for National Action on Clinical Decision Support J Am Med Inform Assoc 2007;14(2):141-145. [PMC free article] [PubMed]
5. CDC. National Center for Health Statistics Electronic Medical Record Use by Office Based Physicians: United States, 2005 2007. Accessed June 30, 2007.
6. Jha AK, Ferris TG, Donelan K, et al. How Common Are Electronic Health Records in the United States?. A Summary of the Evidence. Health Affairs 2006;25:w496-w507. [PubMed]
7. Bridges CB, Thompson WW, Meltzer MI, et al. Effectiveness and cost-benefit of influenza vaccination of healthy working adults: a randomized controlled trial JAMA 2000;284:1655-1663. [PubMed]
8. Gross PA, Hermogenes AW, Sacks HS, Lau J, Levandowski RA. Efficacy of influenza vaccine in elderly persons: a meta-analysis and review of the literature Ann Intern Med 1995;123:518-527. [PubMed]
9. Centers for Disease Control and Prevention Influenza and pneumococcal vaccination levels among persons aged >65 years—United States, 2001 MMWR 2002;51:1019-1024. [PubMed]
10. Morita J. Addressing racial and ethnic disparities in adult immunization, Chicago J Public Health Manag Pract 2006;12:321-329. [PubMed]
11. Centers for Disease Control and Prevention Prevention and control of influenza: Recommendations of the Advisory Committee on Immunization Practices MMWR 2003;52(RR08:1-36. [PubMed]
12. Kawamoto K, Houlihan CA, Balas A, Lobach DF. Improving clinical practice using clinical decision support systems: a systematic review of trials to identify features critical to success BMJ 2005;330:765(2April), doi: (published 14 March 2005).10.1136/bmj.38398.500764.8F [PMC free article] [PubMed] [Cross Ref]
13. Kawamoto K, Lobach DF. Clinical Decision Support Provided within Physician Order Entry Systems: A Systematic review of Features Effective for Changing Clinician Behavior AMIA Annu Symp Proc 2003:361-365. [PMC free article] [PubMed]
14. Miller RA, Waitman LR, Chen S, Rosenbloom ST. The anatomy of decision support during inpatient care provider order entry (CPOE): Empirical observations from a decade of CPOE experience at Vanderbilt J Biomed Inform 2005;38(6):469-485. [PMC free article] [PubMed]
15. Nies J, Colombet I, Degoulet P, Duriex P. Determinants of Success for Computerized Clinical Decision Support Systems Integrated into CPOE Systemes: a Systematic Review AMIA Annu Symp Proc 2006:594-598. [PMC free article] [PubMed]
16. Kaushal R, Shojania KG, Bates DW. Effects of Computerized Physician Order Entry and Clinical Decision Support Systems on Medication Safety Arch Intern Med 2003;163:1409-1416. [PubMed]
17. Dexter PR, Perkins SM, Maharry KS, Jones K, McDonald CJ. Inpatient Computer-Based Standing Orders vs Physician Reminders to Increase Influenza and Pneumococcal Vaccination Rates JAMA 2004;292:2366-2371. [PubMed]

Articles from Journal of the American Medical Informatics Association : JAMIA are provided here courtesy of American Medical Informatics Association