|Home | About | Journals | Submit | Contact Us | Français|
The perfect storm of dramatic cost pressures, increasing public awareness of quality problems, and concerns arising from a significant nursing shortage have escalated hospitals’ attention to improving the efficient management of costly resources. With nursing unit costs comprising a substantial portion of hospital operating expenses, improving the efficiency of nursing care delivery has become a critical strategic goal. Yet evaluations of the efficiency of nursing units have rarely been reported. Hospital managers typically examine profitability, services offered, and patient populations served. The information is then compared with resources consumed to evaluate organizational efficiency. This analysis tends to take place at the hospital level, which ignores the fact that nursing units are significant resource consumers within hospitals. In addition, care provided to patients on different nursing units is sufficiently distinct that aggregation of cost and expense data to the hospital level ignores a rich source of data at the nursing unit level – data that, were it systematically collected and investigated, could provide valuable information regarding potential approaches to improve efficiency.
Economists typically view efficiency as either technical efficiency or allocative efficiency. A firm is technically efficient when it produces the maximum output from a given amount of input, or alternatively, produces a given output with minimum input quantities (Hollingsworth, 2008). A firm is allocatively efficient when the input mix minimizes cost, given input prices, or alternatively, when the output mix maximizes revenue, given output prices. Taken together, technical and allocative efficiency comprise overall efficiency; when a firm is overall efficient, it operates on its cost or revenue frontier (Hollingsworth, 2008).
Traditional hospital efficiency measures have utilized selected ratios in an attempt to understand the relationship between a single output and a single input variable. Unfortunately, the unidimensional nature of this method makes it impossible to consider additional variables or possible environmental influences that may affect efficiency. A more comprehensive approach can be found in regression analysis, because it can accommodate multiple inputs and outputs. However, regression techniques examine “average” practice, when what would be of more substantive interest would be to examine “best” practice. Neither the ratio method nor regression analysis can identify specific sources of inefficiency, nor can either method accommodate the multiple outputs that characterize hospital production. Further, the inability of either ratio methods or regression analysis to identify specific sources of inefficiency leaves a significant gap in the capacity of managers to improve the performance of areas for which they are responsible (McGlynn, 2008; Ozcan, 2008).
Recent developments in management engineering and econometrics have produced techniques that do accommodate the use of multiple inputs and multiple outputs, provide information on sources of inefficiency, and identify best practice work units, thus providing actionable information for managers. In particular, the technique of Data Envelopment Analysis (DEA) is one that has been used to study hospital efficiency. DEA is designed to measure relative efficiency in situations where there are multiple inputs and outputs and there is no obvious objective way of aggregating either inputs or outputs into a meaningful index of productive efficiency (Ozcan, 2008). This method determines a “best practice” frontier that is built empirically from observed inputs and outputs. Each relevant organizational unit, called a “decision-making unit” (DMU), is then compared with its peers. The optimally functioning DMU achieves an efficiency score of 1. DMUs not on the frontier attain efficiency scores between 0 and 1.
DEA has been used extensively in non-health care settings. Since 1983, it has gained popularity as a useful tool for measuring productivity in healthcare settings (Clement, Valdmanis, Bazzoli et al., 2008; Hollingsworth, 2008; Ferrier, Rosko & Valdmanis, 2006; Ozcan, 2008). A recent evidence report on health care efficiency measures completed by the Agency for Healthcare Research and Quality (McGlynn, 2008) cited only 3 studies examining efficiency in nursing, none of which used DEA. However, we found 2 older studies that did use DEA, one of which (Nunamaker, 1983) examined the efficiency of nursing departments, the other (Juras and Brooks, 1993) reported on efficiency at the nursing unit level. In that study, Juras and Brooks found that 32% of the 41 nursing units were relatively inefficient (efficiency scores ranged from .51 to .86), that the use of rotating shifts was negatively associated with efficiency, and the use of acuity data to prepare the staff budget was positively associated with efficiency.
Although DEA has increasingly been used in hospital efficiency analyses, it has only rarely incorporated quality or patient safety variables. In fact, the AHRQ evidence report calls quality measures in efficiency reports “silent” (McGlynn, 2008, p. iv), and recommends that studies of efficiency also contain quality measures. Some newer studies have incorporated quality or patient safety variables as outputs. For example, Clement et al. (2008) found that lower technical efficiency was associated with poorer risk-adjusted quality outcomes, measured with the AHRQ’s Inpatient Quality Indicators (IQIs). And Nayar and Ozcan (2008) found that technically efficient hospitals were performing well with regard to several process quality measures for patients hospitalized with pneumonia. Examining 1,377 urban hospitals, Valdmanis, Rosko and Mutter (2008) found that in low-quality hospitals quality could be improved by increasing labor inputs, in contrast to high quality hospitals, which tended to use too much labor. These recent studies provide examples of the ways in which quality and patient safety variables can be integrated into DEA for a more comprehensive examination of both efficiency and quality of care.
The purpose of our study was to assess the technical efficiency of general medical, general surgical, and combined medical-surgical nursing units, and to incorporate into the DEA model several relevant indicators of quality and patient safety.
DEA is a non-parametric optimization-based technique that constructs an efficiency frontier by maximizing the weighted output/input ratio of each DMU (i.e., the nursing unit), given the constraint that the ratio can equal but never exceed 1 (Chirikos & Sear, 2000; Ozcan, 2008). Nursing units that receive an efficiency score of 1.0 are said to be “optimally efficient” when compared with other nursing units in the sample. These units lie on the efficiency frontier. Nursing units receiving efficiency scores less than 1.0 are operating inefficiently, and the lower the score, the more inefficiently the unit is running. DEA also produces information about ideal targets that help managers to identify where there are excessive inputs or a lack of outputs on inefficient units. Managers can then decide on actions that would need to be taken on that nursing unit to reach the efficiency frontier. It should be noted that the efficiency scores produced by DEA do not represent efficiency in an absolute sense, but rather a unit’s efficiency relative to other nursing units in the sample.
Researchers must choose among several types of DEA models, based on assumptions made about management processes. One distinction is whether the model will be input- or output oriented (Cooper, Seiford & Tone, 2007). Input models focus on the extent to which input quantities can be reduced without changing output quantities, while output models focus on an organization’s attempt to maximize outputs without altering input quantities. Input models assume that managers have control over the inputs, while output models assume managers have control over the outputs. A third type of model, referred to as a non-oriented, slack, or additive model, assumes that managers have control over both inputs and outputs, rather than giving primacy to either (Ozcan, 2008).
Another distinction made between DEA models is whether they assume constant returns to scale (CRS) or variable returns to scale (VRS). The CRS model assumes there is a linear, proportional change in outputs for changes in inputs while the VRS assumes that returns are dependent upon changes in volume. In addition, the CRS model is more stringent than the VRS model since organizations identified as “efficient” in the latter model will always be efficient in the former, but not conversely. The VRS structure is viewed as more appropriate here since the nursing units in the current study vary by size (i.e. number of beds on the unit) (Ozcan, 1992). Thus, we used a VRS model (assuming a unit increase in inputs produces varying increases in outputs) combined with the additive model (assuming that managers have control over both inputs and outputs).
Data utilized for this study were obtained as part of the Outcomes Research in Nursing Administration II (ORNA II) project. ORNA-II is a large, multi-site study designed to investigate relationships among hospital context, structure and organizational, nurse and patient outcomes. Details of the theoretical framework, design, and method have been published elsewhere (Mark, Hughes, Belyea et al., 2007; 2008). After obtaining IRB approval at the University of North Carolina at Chapel Hill and in each participating site, ORNA-II data were collected in 2003 and 2004 on two medical-surgical units in 146 U.S. acute care hospitals that were randomly selected from the 2002 American Hospital Association Guide to Hospitals. Several nursing units withdrew from the study, resulting in a final sample of 286 units in the parent study. Two hundred twenty-six units provided complete data on the variables for the DEA analysis, and are included in this analysis.
Input variables are those that define resources used to produce an output. In general, DEA inputs in health care studies consist of variables representing labor, capital assets and/or other operational expense (Ozcan, 1992). Input variables selected for this study included those that can be readily accessed by nursing unit managers and are used for standard hospital financial reporting purposes.
Labor, the predominant input category, was measured by the number of hours of care per patient day by skill mix level (registered nurse, licensed practical nurse, and unlicensed assistive personnel).
We also included the unit’s operating expenditures. Because we collected data in 2003 and 2004, we inflation adjusted 2003 data by using the Professional Services Index of the CPI-U Medical Care Indices (Mark, Lindley & Jones, 2009). In addition, we adjusted for geographic variability in wages by using the Centers for Medicare and Medicaid Services (CMS) wage index. Our measure was thus wage and inflation adjusted operating expenses for the unit (in $000,000).
Capital assets, commonly defined as plant, equipment and support technology, were not explicitly measured in the ORNA project, in part because of the variation in accounting practices across hospitals. Instead, we used the number of beds on the unit as a proxy measure to capture unit level capital assets (Ozcan, 2008; Rosko, 1990).
Four outputs were included in this study based upon their appropriateness to the evaluation of nursing unit efficiency and their easy availability to managers. The first was the number of discharges from the unit. Discharges represent the productivity-based output measure. To facilitate comparisons of resource use across nursing units, and to take into account a potentially important source of difference in units’ levels of efficiency, we adjusted the number of discharges by the CMS case mix index.
The other three outputs represent the quality-based and patient safety output measures. The first of these was patient satisfaction, our “overall” measure of quality, which was derived from a survey completed by a random sample of 10 patients on each nursing unit. Patients rated thirteen items on a four-point scale ranging from poor to excellent. Sample items included the ability of caregivers to work together, the promptness of nurses in answering calls, and the quality and thoroughness of the nursing care. Patients were also asked to rate on a four-point Likert-type scale ranging from never to always, the extent to which they perceived respect and dignity from nurses, if they considered the hospital a place for good nursing care, and whether they would be comfortable recommending the hospital to a friend or relative. Cronbach’s alpha for the scale was 0.97.
In addition, we also included variables more representative of patient safety -- the numbers of reported medication errors and patient falls. Medication errors were defined as an error in medication administration (wrong patient, drug, dose, time, or route); falls were defined as any unplanned descent to the floor. These variables were measured using incident report data with values scaled to 1000 inpatient days. We used the reciprocal of these values so that fewer adverse events represent the more desirable outcome. Thus, these four output variables incorporated dimensions of productivity (discharges) and quality of care (patient satisfaction), and patient safety (medication errors and falls). Table 1 depicts the variables from the ORNA II data set used in the current analysis.
Data collection continued for six consecutive months in each hospital. Each hospital appointed an on-site study coordinator to assist with data collection. All study coordinators participated in a 1 1/2 day training program conducted by the ORNA-II research team. This program provided detailed information about the purpose and objectives of the study and procedures for data collection. Several steps were taken to ensure data integrity. First, all study coordinators were given a procedure manual that included the information presented during the training program. Second, all data were immediately reviewed by the research team on arrival in the research office, and study coordinators were contacted by telephone, fax, or E-mail to resolve data discrepancies. Finally, calculations required for the measurement of selected variables were completed in the research office to ensure that the same formulae were used and that mathematical errors were identified and corrected. Study coordinators provided data on the input variables of RN hours, LPN hours, unlicensed hours, operating expenses and the number of beds on the unit. They also provided data on the unit’s number of discharges, number of medication errors and number of patient falls.
In addition, 10 patients on each unit aged 18 years or older, hospitalized for at least 48 hours, who were also able to speak and read English, and who were not scheduled for immediate discharge were randomly selected to complete a questionnaire about their satisfaction with nursing care. A total of 2213 patients (93% response rate) replied to the questionnaire.
Data were analyzed with the DEA Excel Add-on Solver (Zhu, 2003). Descriptive statistics of the model variables are shown in Table 2. On average, nursing units had 34 beds, and provided 5.09 RN hours of care 0.75 LPN hours of care, and 2.31 hours of care per day by unlicensed assistive personnel. In terms of outputs, units had 1600 adjusted discharges over the six-month period. Patients were generally satisfied with the care they received, reflected in the mean patient satisfaction score of 3.43 on a 5-point scale. On average, there were 2.56 medication errors per 1000 patient days and 2.94 patient falls per 1000 patient days.
Results describing the number of efficient and inefficient units identified from the VRS additive (non-oriented) model are shown in Table 3. Although 65 (40.4%) nursing units were located on the efficiency frontier, 161 (59.6%) units were assessed as being inefficient. Average efficiency of the inefficient units was 0.231 on a 0-1 scale. Overall efficiency (including both efficient and inefficient units) was 0.452.
Table 4 provides the results for the entire set of 161 inefficient units in the sample. Because we used a “non-additive model,” i.e., one that assumes managerial control over inputs as well as outputs, and we included both productivity and quality and patient safety outputs, this information takes into account the possibility that poor quality or poor patient safety may contribute to inefficient performance. The column titled CURRENT VALUE provides the actual value of each of the input and output variables for the inefficient units (on average); the next column (TARGET VALUE) provides the value for each input and output variable that would place the unit on the efficiency frontier. The last column indicates the direction of the change, the amount of the change, and the percentage change to be accomplished to place the unit on the efficiency frontier. Our results suggest that all inputs need to be reduced for the inefficient units to improve to benchmark levels of efficiency. The input with the least change needed is RN hours of care (which needs to be decreased by 0.27 hours [or approximately 16 minutes]), while the largest reduction would be in LPN hours, which, when measured as a percentage is a 52.9% reduction, is actually a small number of minutes (from 45 minutes of care to 21 minutes of care provided by LPNs). On the output side, although small increases in discharges and patient satisfaction are called for, the analysis suggests that major, and very substantial, improvements in patient safety are necessary to achieve the efficiency of the benchmark units in the sample. The greatest change needed is in reducing medication errors but patient falls need to be reduced substantially as well. These results clearly demonstrate the importance of patient safety in achieving optimal efficiency.
Data envelopment analysis also produces information on each DMU (nursing unit) that can assist managers to identify specific aspects of unit performance where improvements can be considered. To illustrate this, we more or less randomly selected one of the poor performing units, Unit #312, which had an efficiency score of 0.462. Table 5 displays these results. As can be observed, compared to benchmark (i.e., efficient) units, Unit #312 should reduce the number of beds on the unit from 49 to 29.5, a reduction of 34.2%, and reduce LPN hours by 73.7 percent from 2.42 hours to 0.63 hrs. Similarly, unlicensed hours require approximately a 50.8% reduction – from 2.76 hours per patient day to 1.36 hours. However, no change is needed in RN hours or operating expenses. On the output side, medication errors and patient falls need to be reduced by 43.8% and 78.4 %, respectively. Furthermore, this unit needs to improve average patient satisfaction by 25.6%, from 2.9 to 3.64 (on 1-5 scale). Although we present the results for only a single unit, it is important to note that DEA provides this information on every unit in the sample, thus allowing for a unique improvement approach to be developed for each unit.
In this analysis, we discovered that nearly 60% of nursing units in our sample were operating inefficiently. Interestingly, traditional targets for efficiency improvements, such as providing fewer hours of nursing care and reducing operating expenses, are not managerial strategies that would yield the largest efficiency gains. Our analysis suggests that it is in the reduction of medication errors and patient falls – two critical patient safety targets – where the greatest efficiency gains are likely to be seen. In the current health care environment, where there is an intense focus on cost, our results, which highlight the primacy of quality and patient safety in producing efficiency, are startling.
To investigate whether there were other important differences between efficient and inefficient units that might have contributed to our results, we compared the units on several key variables that have been associated with nursing unit performance (Mark, Hughes & Belyea et al., 2007; 2008; Mark, Lindley & Jones, 2009). In a supplemental analysis (not shown, but available upon request from the first author), we found no statistically significant differences between efficient and inefficient units on urban versus rural location, Medicare case mix index, the number of hospital beds, Magnet certification, the proportion of RNs on the unit with a bachelors degree, RNs’ length of experience, or the availability of a variety of support services on the nursing unit. Although this supplemental analysis does not reveal any obvious confounding in our findings, they still should be taken cautiously because we do not have any information on unit level micro-processes, such as model of care delivery, specific staffing patterns, architectural layout or the availability and extent of electronic medical record and other computerized patient systems. These could certainly be important factors in our output measures of productivity (adjusted discharges), quality (patient satisfaction) and patient safety (medication errors and patient falls).
There are additional limitations inherent in the methodology of DEA that should be kept in mind. First, DEA reflects relative efficiency, not absolute efficiency. Those DMUs (nursing units) in the sample that are operating at the “highest” level of efficiency compared to their peers are designated as “efficient.” However, DEA does not allow us to evaluate whether they are operating “efficiently” according to an external gold standard. Second, the selection of inputs and outputs can be controversial, and the results from the DEA are entirely dependent on those selections. We selected input and output variables primarily based on their likely contribution to unit efficiency, determined partly from a review of earlier hospital-level DEA studies, but also because these variables are generally easily available to unit managers. However, other combinations of inputs and outputs would likely yield different results. Finally, because of the highly quantitative nature of DEA results, it may be tempting to design improvement strategies that target directly the areas identified by the results as needing attention. Certainly, a manager receiving information indicating that medication errors need to be reduced by nearly 750% is likely to want to take immediate action. However, the DEA results should instead be used as a starting point for a conversation with staff members to identify key points in the production process where improvements may occur, taking into account the strategic goals of the particular unit and hospital, the unit’s workforce and the nature of the patients cared for.
The goal of this study was to assess the technical efficiency of hospital-based nursing units. This study is one of few to have explored a non-parametric method to assess nursing efficiency. Future research in nursing efficiency might compare different methodologies such as regression, ratio analysis, and DEA, or investigate alternative inputs and outputs. Using an alternative methodology to examine efficiency, our quality and patient safety findings suggest a different approach to managing inefficiency in contrast to the traditional means of reducing labor and operating expenses. In today’s health care environment where nursing staff levels are chronically low because of shortages and reimbursement methods have forced health care organizations to make operational cuts, this study imparts new and practical information to managers as they work to improve efficiency. The results of our study suggest that better understanding the drivers of quality and patient safety can contribute substantially to improvements in nursing unit efficiency.
Although our study included only 2 medical-surgical nursing units in each hospital, DEA can be used to examine multiple nursing units; in fact, a hospital could include all nursing units in a larger analysis. This would allow more extensive comparisons among similar and different types of units as well as provide a basis for organizational decision-making about how best to achieve overall hospital nursing efficiency. Inefficiency implies waste, whether it arises from less than optimal use of input resources or from poor quality, and such waste is a major component of excess health care costs. In the current economic environment, hospital will likely seek out new ways of targeting areas for potential cost savings since past approaches have been only marginally successful. While remaining cognizant of its limitations, DEA can be used to better understand the complex interrelationships among efficiency, quality, and patient safety in the delivery of acute care nursing. From a policy perspective, such information can provide valuable insights in the current debate about possible structural and payment alternatives in a reformed health care system.
The research reported here was supported by grant number 5R01NUR003949 from the National Institute of Nursing Research, National Institutes of Health