Search tips
Search criteria 


Logo of jamiaAlertsAuthor InstructionsSubmitAboutJAMIA - The Journal of the American Medical Informatics Association
J Am Med Inform Assoc. 2011 May-Jun; 18(3): 322–326.
Published online 2011 March 21. doi:  10.1136/amiajnl-2010-000066
PMCID: PMC3078659

Improving the utilization of admission order sets in a computerized physician order entry system by integrating modular disease specific order subsets into a general medicine admission order set


Case description

We evaluated the effects of integrating order subsets for the most common medical diagnoses into a general medical admission order set of our electronic medical records (EMR) in order to improve order set integration by clinicians.

Methods of implementation

We identified the most common primary and secondary diagnoses for patients admitted to our medical service and developed order subsets comprising only of the orders necessary for the management of these individual diagnoses. Using the capabilities of our computerized physician order entry (CPOE), we nested these order subsets into the general order set and evaluated the resulting change in order set utilization by our clinicians.

Example and observations

The total number of order sets used by clinicians in all departments increased fivefold during the 16-month period following the implementation of the integrated order sets in July 2008. A before and after time series was used to analyze the trend in increased order set usage and showed an effect of the intervention (p=0.023).


Integration of disease specific order subsets into a single general admission order set significantly improved the overall adoption of order sets by clinicians. This provides health care systems with the opportunity to improve patient safety and implement evidence based care in clinical practice.

Keywords: Computerized physician order entry, electronic medical record, standardized order, nested order, usual, CPOE, improving the utilization of admission order sets, machine learning

Case description

General and disease specific admission order sets are used by many hospitals. They are expected to reduce the time and effort needed to place common orders and reduce the possibility of missing important orders that improve patient safety and compliance with core measures related to quality of patient care. Previous studies have demonstrated that such admission order sets improve immunization rates and appropriate antibiotic use and probably reduce length of stay in hospitalized patients.1

We describe our experience with increasing electronic admission order set use by integrating disease specific order subsets into a standard admission order set in the medicine service of a busy academic medical center.

Sinai-Grace Hospital (SGH) is a 404-bed hospital with 700 physicians, 670 nurses, and more than 1800 support staff. SGH discharges more than 22 500 patients per year and its emergency department is among the busiest in Detroit, treating more than 84 000 patients a year.

The hospital uses the Powerchart electronic clinical information system developed by Cerner Corporation as an electronic medical records (EMR) system for documentation of clinical notes, diagnostic studies, and physician orders. Prior to the implementation of the EMR, paper based general admission order sets had been introduced but the utilization of these order sets was low.

Both paper based and electronic order sets have limitations. Similar to the inherent shortcomings of paper based order sets,2 lack of national guidelines for different issues such as computerized physician order entry (CPOE) based drug–drug interaction information, checking or recording of drug allergies electronically,3 implementation without organization standards, and clinical review, or not being properly maintained present a challenge for electronic order sets adoption.2 However, CPOE has been shown to have potential in reducing medical errors4 and costs,5 and improving the quality of care.6

At SGH, as in other organizations,7 8 evidence based order sets were developed with considerable input from specialists to ensure compliance with disease management guidelines and best practice, but the ability to accomplish these objectives was constrained by the low order set use by clinicians. We did not conduct a user satisfaction survey,9 but informal discussions with our staff showed high levels of dissatisfaction at the initial implementation of the order sets. Another challenge was that, although our patients were admitted with a principal diagnosis that required the use of the admission order set, management of several other conditions required the use of extra order sets or writing out additional orders using a regular order sheet.

Methods of implementation

The hospital implemented CPOE in May 2007 as part of the transition to the EMR. Prior to the implementation, all paper order sets were reviewed and equivalent electronic order sets developed and placed in a medicine order set catalog. The conversion to electronic order entry was implemented over a 24–48 h period and all paper based order entry discontinued shortly afterward. The most commonly used orders were pre-selected and personal order sets were also permitted. All medical staff received training in CPOE.

Despite several educational programs and repeated reminders, the use of the electronic admission orders sets remained low. An informal survey of house staff indicated that the utility of these order sets was limited because many of the patients admitted to the medicine service had several other conditions that required concurrent management during hospitalization. Clinicians preferred to build their own set of admitting orders rather that use one or more disease specific order sets. In order to address this need, a program was developed that allowed the disease specific order subsets to be integrated into a general order set that could be called up by the clinician admitting the patient based on the applicable diagnoses. These disease specific order subsets were different from the disease specific order sets because general admission orders such as diet, activity, and routine laboratory tests were not repeated within the subsets. The clinician would build the complete set of admitting orders relevant for the patient in a modular fashion by starting out with the general order set and calling up the disease specific order subsets in a stepwise manner.

This program was started in June 2008 with a complete analysis of the most common primary and secondary diagnoses of patients admitted to our hospital. Based on this information, the modular order subsets were developed using existing electronic and paper order sets with input from the respective specialists. For the initial implementation, disease specific orders were integrated into the general admission order set and named the comprehensive medical admission order set. The program was first implemented for the academic service in July 2008 and refined based on input from its users. After completing a few modifications based on clinician feedback, hospital-wide implementation was carried out in October 2008 and order set use was monitored using the reporting capabilities of the EMR.

Example and observations

A screenshot of the comprehensive general admit order set with the option to select the nested subsets based on patient co-morbidities is shown in figure 1.

Figure 1
Screenshot of the comprehensive medicine admission order set showing its nested components.

The number of electronic order sets used during the 16-month period following the implementation of the comprehensive medicine admission order set is shown in table 1 (July 2008–October 2009).

Table 1
The number of electronic order sets used for different diagnoses after converting to computerized physician order entry

The total number of electronic order sets used each month increased from 349 in July 2008 to 1711 by October 2009. The number of newly implemented comprehensive nested medicine admission order sets used increased from 74 per month to 1476 per month during this period, almost entirely accounting for the fivefold increase in electronic order set use at our institution.

A before and after time series was used to analyze the trend in increased order set usage. A total of 30 monthly observations of order set usage were examined from May 2007 (when CPOE was first implemented) to October 2009. Analyses showed an effect of the intervention (p=0.023). We also developed an autoregressive integrated moving average (ARIMA) time series model adding a dummy variable that explained about 18% of the variation in the series, while the Ljung-Box statistic had a p value of 0.666, reinforcing the fact that the model was correctly specified (figure 2).

Figure 2
Time series analysis.

The impact of the comprehensive medicine admission order set on the use of disease specific order sets is best illustrated by the analysis of order set usage for patients admitted for non-acute coronary syndrome chest pain (figure 3). The use of order sets was comparatively high for this diagnosis even before the implementation of the comprehensive medicine admission order set, with the stand alone order set for chest pain used 151 times in July 2008. The use of this stand alone chest pain order set decreased to 30 by October 2009, while the use of the nested chest pain orders in the comprehensive medicine order set increased to 213 order sets per month.

Figure 3
Comparison of the nested and non-nested numbers of order sets for non-acute coronary syndrome chest pain.

A similar increase in nested order sets included in the comprehensive medicine order set was noted for other diagnostic entities (figure 4). The order set for community acquired pneumonia (CAP) showed the lowest utilization, while the comprehensive medicine admission order set without any of its nested components was used most often.

Figure 4
Number of nested order sets within the complete medical order sets for the Internal Medicine Department. CAP, community acquired pneumonia; COPD, chronic obstructive pulmonary disease.


The typical adult patient admitted to the medical service of an acute care hospital in the United States has more than one diagnosis that requires concurrent management, especially for older patients having multiple co-morbidities.10 11

Standard admission order sets are being used with increasing frequency and serve as a checklist to reduce the inadvertent omission of orders important for patient care, meeting key performance measures, and promoting best practice.12 The experience with adoption of standard admission orders by healthcare providers seems to vary by diagnosis and complexity of care.13 14 In a study conducted by McAlearney et al among pediatric patients, order set utilization rates were higher for diagnoses such as asthma (88%) and appendicitis (80%) but considerably lower for CAP.13 Our experience with adopting adult medicine admission order sets was even less encouraging prior to the implementation of the integrated order set, probably due to the multiple patient co-morbidities.

During the period that paper based admission order sets were implemented at our hospital, low adoption rates were attributed to the logistics of distributing, cataloging, and stocking these paper order sets throughout the hospital with inconvenient availability to the clinical staff. However, this limitation ceased to be a factor after converting to CPOE, after which immediate access to the entire catalog of standard admission order sets was made possible via a simple drop down menu. As others have found,13 15–17 the continued under-utilization of these order sets after transition to EMR indicates that factors other than physical access to order sets contributed to clinician preference in using this utility.

During the planning phase of this project, we learned a few lessons by identifying several factors contributing to clinician dissatisfaction with this procedure:

  1. Many medical patients had more than one diagnosis at the time of admission and orders for these secondary diagnoses had to be placed separately or by selecting additional order sets for these diagnoses. When a second or third order set is used for this purpose, the clinician has to review and leave out duplicate orders that have already been entered as part of the first order set.
  2. It is also important to strike a balance between ensuring the medical order set is complete by including the common orders but not overburdening the user with an overwhelming choice of selections that are meant to address all eventualities.
  3. In designing this modular, subset based, comprehensive medical order set for our hospital, we tried to minimize these factors by using the general admission order set to include the most commonly needed orders and avoided repeating these orders in the disease specific subsets. Orders necessary to comply with best practice such as adult immunization, smoking cessation counseling, and pain management were all included in this set. Additionally, measures to reduce complications related to hospitalization such as venous thromboembolism and hospital acquired infections, were also incorporated. Order subsets for common co-morbidities such as hypertension, diabetes, and congestive heart failure (CHF) have been specifically designed to meet some of the best practice standards such as basal-bolus administration of insulin for improved glycemic control, hemoglobin A1c measurements, and treatment with ACE inhibitors for patients with CHF.
  4. We also found that use of the nested order subsets for CAP, stroke, and end-stage renal disease did not increase to the same extent as for other diagnoses. In the case of CAP and stroke, this is likely a result of the need to initiate the major portion of the orders for laboratory studies, imaging, and medication administration within very specific time limits, frequently when the patient is still in the emergency department.

The principal limitation of this descriptive study is that we are not able to establish that the use of standardized order sets resulted in improved patient outcomes and cost savings to the institution. Also, this is not a randomized trial, so other factors might have contributed to the increased order set usage. While a comparison of key indicators of patient outcome and cost of care among patients admitted with and without the use of order sets is feasible, random assignment was difficult due to the large number of clinicians involved in patient care. We did not have the necessary resources to conduct a randomized study and collect objective outcome data on our patients to determine if the increased order set use resulted in meeting treatment goals and better care, and we were not able to present data on the percentage of patients for whom the condition specific order sets might be appropriate.

Our strategy to use the functionality of CPOE of an EMR by designing a dynamic modular order set that can be customized to meet the specific needs of the patient being admitted has been well received by the majority of the clinicians at our hospital. Although there are limitations in regards to CPOE use such as time needed to learn the system and difficult user–system interactions,18 current evidence indicates that CPOE systems have the potential to reduce costs and improve the quality of care for our patients.1 16 19 20 Our paper describes a strategy to increase appropriate use of CPOE, as its benefits are unlikely to be achieved if CPOE is not used.


We would like to thank Dr Joann Riggins-Woodhouse and Mrs Rita Miller from the Electronic Medical Records Clinical Transformation team at Sinai-Grace Hospital, Detroit Medical Center, for their assistance with this project.


Competing interests: None.

Provenance and peer review: Not commissioned; externally peer reviewed.


1. Fishbane S, Niederman MS, Daly C, et al. The impact of standardized order sets and intensive clinical case management on outcomes in community-acquired pneumonia. Arch Intern Med 2007;167:1664–9 [PubMed]
2. Bobb AM, Payne TH, Gross PA. Controversies surrounding use of order sets for clinical decision support in computerized provider order entry. J Am Med Inform Assoc 2007;14:41–7 [PMC free article] [PubMed]
3. Gross PA, Bates DW. A pragmatic approach to implementing best practices for clinical decision support systems in computerized provider order entry systems. J Am Med Inform Assoc 2007;14:25–8 [PMC free article] [PubMed]
4. Bates EW, Leape LL, Cullen DJ, et al. Effect of computerized physician order entry and a team intervention on prevention of serious medical errors. JAMA 1998;280:1311–16 [PubMed]
5. Evans RS, Pestotnik SL, Classen DC, et al. A computer-assisted management program for antibiotics and other antiinfective agents. N Engl J Med 1998;338:232–8 [PubMed]
6. Tierney WM, Miller ME, Overhage JM, et al. Physician inpatient order writing on microcomputer workstations. Effects on resource utilization. JAMA 1993;269:379–83 [PubMed]
7. Payne TH, Hoey PJ, Nichol P, et al. Preparation and use of preconstructed orders, order sets, and order menus in a computerized provider order entry system. J Am Med Inform Assoc 2003;10:322–9 [PMC free article] [PubMed]
8. Ozdas A, Speroff T, Waitman LR, et al. Integrating “best of care” protocols into clinicians' workflow via care provider order entry: impact on quality-of-care indicators for acute myocardial infarction. J Am Med Inform Assoc 2006;13:188–96 [PMC free article] [PubMed]
9. Lee F, Teich JM, Spurr CD, et al. Implementation of physician order entry: user satisfaction and self-reported usage patterns. J Am Med Inform Assoc 1996;3:42–55 [PMC free article] [PubMed]
10. Lezzoni LI, Foley SM, Daley J, et al. Comorbidities, complications, and coding bias: does the number of diagnosis codes matter in predicting in-hospital mortality? JAMA 1992;267:2197–203 [PubMed]
11. Boyd CM, Darer J, Boult C, et al. Clinical practice guidelines and quality of care for older patients with multiple comorbid diseases: implications for pay for performance. JAMA 2005;294:716–24 [PubMed]
12. Haynes AB, Weiser TG, Berry WR, et al. A surgical safety checklist to reduce morbidity and mortality in a global population. N Engl J Med 2009;360:491–9 [PubMed]
13. McAlearney AS, Chisolm D, Veneris S, et al. Utilization of evidence-based computerized order sets in pediatrics. Int J Med Inform 2006;75:501–12 [PubMed]
14. Ahmad A, Teater P, Bentley TD, et al. Key attributes of a successful physician order entry system implementation in a multi-hospital environment. J Am Med Inform Assoc 2002;9:16–24 [PMC free article] [PubMed]
15. Eslami S, de Keizer NF, Abu-Hanna A. The impact of computerized physician medication order entry in hospitalized patients-a systematic review. Int J Med Inform 2008;77:365–76 [PubMed]
16. Bogucki B, Jacobs BR, Hingle J. Computerized reminders reduce the use of medications during shortages. J Am Med Inform Assoc 2004;11:278–80 [PMC free article] [PubMed]
17. Rothschild AR, Lehmann HP. Information retrieval performance of probabilistically generated, problem-specific computerized provider order entry pick-lists: a pilot study. J Am Med Inform Assoc 2005;12:322–30 [PMC free article] [PubMed]
18. Niazkhani Z, Pirnejad H, Berg M, et al. The impact of computerized provider order entry systems on inpatient clinical workflow: a literature review. J Am Med Inform Assoc 2009;16:539–49 [PMC free article] [PubMed]
19. Taylor R, Manzo M, Sinnett M. Quantifying value for physician order entry systems: a balance of cost and quality. Healthc Financ Manage 2002;56:44–8 [PubMed]
20. Hulgan T, Rosenbloom ST, Harvgrove F, et al. Oral quinolones in hospitalized patients: an evaluation of a computerized decision support intervention. J Intern Med 2004;256:349–57 [PubMed]

Articles from Journal of the American Medical Informatics Association : JAMIA are provided here courtesy of American Medical Informatics Association