Search tips
Search criteria 


Logo of nihpaAbout Author manuscriptsSubmit a manuscriptHHS Public Access; Author Manuscript; Accepted for publication in peer reviewed journal;
J Nurs Adm. Author manuscript; available in PMC 2017 March 7.
Published in final edited form as:
PMCID: PMC5340260

Quality Outcomes of Hospital Supplemental Nurse Staffing

Ying Xue, DNSc, RN, Linda H. Aiken, PhD, RN, FAAN, Deborah A. Freund, PhD, and Katia Noyes, PhD



Use of supplemental registered nurses (SRNs) is common practice among U.S. hospitals to fill gaps in nurse staffing.


To examine the relationship between use of SRNs and patient outcomes.


Multilevel modeling was performed to analyze hospital administrative data from 19 hospital units in a large tertiary medical center for the years 2003–2006. Patient outcomes included in-hospital mortality, medication errors, falls, pressure ulcers, and patient satisfaction with nurses.


SRN use ranged from 0–30.4% of total RN hours per unit quarter. Among 188 of the 304 unit quarters in which SRNs were used, the average SRN use was 9.8% in non-ICUs and 6.4% in ICUs. All observed effects of SRN use on patient outcomes were non-significant.


SRN use was substantial and varied widely by unit. No evidence was found that links SRN use to either adverse or positive patient outcomes.


Hospitals use supplemental registered nurses (SRNs) hired from staffing agencies to fill vacant nursing positions temporarily. These SRNs work either as per diem or traveling nurses with multiple-month contracts. In 2001, 56% of U.S. hospitals used traveling nurses1. In 2006, the community tracking study, monitoring 12 U.S. geographic markets, indicated that 75% of participating hospitals used agency nurses as supplemental nursing staff2. Among hospitals using SRNs, the level of use varied across hospital unit types, with a range of between 0–33%3.

Despite the widespread use of SRNs, only a few empirical studies have investigated the effects of SRN use on patient outcomes such as mortality, medication errors, central line infections, catheter associated bloodstream infections, and falls in hospitals. The findings from these studies have been inconsistent. One possible reason is that some studies were not designed to examine the relationship between use of SRNs and patient outcomes but included SRN use as a covariate45. The lack of a theoretical match could obscure the relationships between use of SRNs and patient outcomes. Another possible reason is that most studies did not adjust for the potential confounding effect of the nursing work environment (NWE) on the association between SRN use and patient outcomes67. The NWE has been recognized as a key factor in the promotion of quality patient care. Empirical evidence has demonstrated that better NWEs are linked to higher nurse-to-patient ratios and better patient outcomes89, and poorer NWEs are linked to poorer patient outcomes, as well as higher nurse burnout, job dissatisfaction and nurse turnover8. Nurses who work in less supportive work environments are more likely to leave their hospital8, resulting in vacant positions, creating the need for SRNs2. One notable exception is the study by Aiken, Xue, Clarke, & Sloane3, in which poorer NWE with inadequate staff and not the use of SRNs were found to explain negative patient outcomes at the hospital level. This study demonstrates the critical need to consider the nature of NWEs when examining the relationship between use of SRNs and patient outcomes. Another limitation of previous studies is the absence of analysis involving SRN use and patient outcomes at the hospital unit level over time. Levels of SRN use, patient outcomes, and NWEs vary widely across units even in the same hospital10. Unit-level analysis would capture these variations and strengthen the estimation of the effect of SRN use on patient outcomes.

Building on the limited empirical evidence of the previous studies, the objective of this study was to examine the relationship between use of SRNs and patient outcomes using unit-level data, while controlling for features of the NWE and other covariates. Patient outcomes that were examined included in-hospital mortality, medication errors, falls, pressure ulcers (PU), and patient satisfaction with nurses. These outcomes have been recognized as key quality indicators1112. Our study was limited to SRNs who were hired through several staffing agencies and had 3-month renewable contracts because the study hospital from which data were collected exclusively employed this type of SRN.


Research Design

A retrospective multilevel longitudinal design was used to analyze hospital administrative data from 19 hospital units, including 15 adult medical, surgical or step down units and 4 intensive care units (ICUs) within a large tertiary medical center in the Northeast. Retrospective hospital data were collected for the years 2003–2006, the most current available data at the time of data collection. The hospital received Magnet designation in 2004. The study was approved by the university research subjects review board.

Data were collected quarterly from each of the 19 hospital units from 2003 to 2006, resulting in a total of 304 data collection points (19 units × 16 annual quarters per unit). Multilevel modeling was used to account for multiple observations over time by annual quarter nested within units. Our multilevel analysis thus consisted of 2 levels. The 1st level (level-1) consisted of multiple observations over time by annual quarter within units. Level-1 variables were those that changed across quarters within units and included patient characteristics, availability of rapid response team (RRT, used only with the mortality outcome model), nurse staffing characteristics and patient outcomes. The 2nd level (level-2) was composed of unit characteristics that were constant over time within each unit but varied across units. Level-2 variables included type of unit and NWE.

The multilevel modeling design enabled separation of between (level-2) and within (level-1) hospital unit variance13. Partitioning the between-unit differences from the remainder of the variance improved estimation of effects within units13 and may therefore reveal the effect of SRN use on patient outcomes, which might be less apparent in a study using only one level of data.

Variables and Measures

We obtained data on discharge patients, nurse staffing and unit characteristics. Variables, operational definitions, frequency of measurement, and level of data for multilevel modeling are listed in Table 1. All level-1 variables were measured as aggregate data by annual quarter by hospital unit between 2003 and 2006. The NWE was measured as a level-2 variable based on survey data collected in 2003 and 2005.

Table 1
Variables and their operational definitions, frequency of measurement and level of data for multilevel model

Outcome Variables

In-hospital mortality, medication errors, and falls were obtained from a hospital office that maintained clinical and national comparative databases to support and facilitate outcomes research and local quality improvement efforts. Medication errors and falls were reported by clinicians through the hospital’s voluntary electronic reporting system, an approach that has been used to improve error reporting nationally14. Data on PUs were extracted from the hospital database reported to the National Database of Nursing Quality Indicators (NDNQI). Patient satisfaction with nurses was measured by the mean of the subscale of the nurses section of the Press Ganey patient satisfaction survey15. Satisfaction scores ranged from 0 to 100, with a higher score representing higher satisfaction.

Primary Predictor

The primary predictor was use of SRNs. All SRNs were registered nurses (RNs). SRNs did not float between units but worked for the same unit under a contract. We defined use of SRNs as the proportion of total RN hours provided by SRNs per unit per quarter; this proportion measure has been used in prior studies3, 16.


Covariates included patient characteristics, availability of rapid response teams (RRTs) (used only with the mortality outcome model), nurse staffing characteristics (other than use of SRNs) and unit characteristics.

Patient Characteristics Used As Level-1 Covariates

The set of covariates used for risk adjustment in each outcome model was selected based on theory, extant literature and availability of data. For the outcome of pressure ulcers, we adjusted for patient age, gender, and pressure ulcer risk as measured by the Braden scale17. For the outcome of patient satisfaction with nurses, we adjusted for patient age and gender. For the outcomes of in-hospital mortality, medication errors, and falls, we adjusted for patient age, gender, race/ethnicity, primary source of payment, principal diagnosis, diagnosis-related group service intensity weight, and comorbid conditions. Comorbidity was measured by the Charlson/Deyo comorbidity score18, which was calculated based on up to six secondary diagnoses.

Availability Of RRT Used Only As A Level-1 Covariate With The Mortality Outcome Model

During the study period, the hospital was in the process of implementing hospital-wide RRT on different units at different times. Use of RRT has been shown to reduce preventable deaths19. We therefore controlled for the availability of RRT for the mortality outcome. Since availability of RRT varied over time within each unit, it was treated as a level-1 covariate.

Nurse Staffing Characteristics (Other Than Use Of SRNs) Used As Level-1 Covariates

Prior research has shown that adequate hospital nurse staffing was associated with better patient outcomes2021. We controlled for nurse staffing level in the analysis, measured by total nursing hours per patient day. In addition, we also controlled for the effects of nurse educational level, nursing experience, and RN skill mix because prior studies have shown a link between these variables and patient outcomes20. Since there is no evidence to indicate a difference in the effects of these attributes on patient outcomes between permanent nurses and SRNs, we modeled nurse educational level and nursing experience by pooling the data of both permanent nurses and SRNs.

Unit Characteristics Used As Level-2 Covariates

Unit characteristics included type of unit (ICU vs. non-ICU) and nursing work environment. The 19 units included 4 ICUs and 15 non-ICUs that consisted of 4 medical, 7 surgical and 4 step-down units. Palliative care units were not included. Dichotomization of unit type to ICU and non-ICU categories was based on preliminary analyses that showed no difference in outcomes among medical, surgical and step-down units (non-ICUs) compared to ICUs.

Nursing work environment was measured using the NDNQI adapted index of work satisfaction (IWS), which the hospital had selected to survey nurses. The psychometric properties of the adapted IWS have met standards for reliability and validity in a wide variety of hospital settings and RN populations. The range of internal consistency reliability has been reported as 0.63 to 0.91 for the 7 subscales22. Theoretically, according to Lake23, the concept of NWE consists of 7 domains, 5 of which match subscales of the IWS. The IWS has been shown to correlate highly (r=0.68) with the National Quality Forum-endorsed practice environment scale24, a frequently used measure of NWE, which covers 4 of the 7 domains.

In order to better differentiate NWE among units within the hospital, we used the modified IWS scores to create a 3-category variable indicating better, mixed, and poorer work environment. This classification has been widely used in prior research and has demonstrated good predictive validity8. Since preliminary analysis showed that measures of nursing work environment did not vary significantly over time in each unit, we coded NWEs as better, mixed, or poorer for each unit based on the average scores of years 2003 and 2005 and treated it as a level-2 covariate.

Statistical Analysis

Data were aggregated by annual quarter within each hospital unit and hierarchically structured into the 2 levels. The unit of analysis at level-1 was the unit quarter; data consisted of 16 annual quarters from 2003 to 2006 nested within each of 19 hospital units, for a total of 304 observations. The unit of analysis at level-2 was the hospital unit, of which, as noted, there were 19.

Prior to inferential analysis, we performed diagnostic tests to ensure no violations of the assumptions of multilevel modeling. Using 2-level multilevel modeling, we modeled intercepts as random effects in order to examine the effects of level-2 predictors25. We used restricted maximum likelihood for estimating the statistical parameters for the random intercept models25. Unconditional models were evaluated first for each outcome variable separately. The intraclass correlation coefficients (ICC) for all outcome variables were significant at alpha 0.05 with a range of 0.33 to 0.90, indicating the need for multilevel analysis. Next, we performed multilevel analyses by entering all level-1 and level-2 predictors to specify each of the outcome models. Bias-corrected bootstrap methods were used to generate model parameter estimates. These methods have been shown to produce more accurate estimates in multilevel modeling when level-2 sample size is small or distributions deviate from normal26. All reported p-values were adjusted using the false discovery rate method for type I experiment-wise error for multiple comparisons27. Analyses were performed using SAS version 9.2. All statistical tests were 2-sided, with alpha set at 0.05.


Use of SRNs in the Hospital

Use of SRNs (proportion of total RN hours provided by SRNs) ranged from 0–30.4% per unit quarter. SRNs were used in 188 of the 304 unit quarters observed (61.8%). Among the 188 unit quarters in which SRNs were used, the average SRN use was 9.8% (SD=7.2%) in non-ICUs and 6.4% (SD=4.7%) in ICUs. In SDC #1, we present SRN use by unit among quarters when SRNs were used. One unit (#12) did not use SRNs at all and 1 unit (#19) used SRNs in all 16 quarters. Other units used SRNs between 3–15 quarters. Among the 18 hospital units that used SRNs, the range of SRN use was from 0.2 to 30.4%. Unit 7 had the widest range of use (1.5%–30.4%) and unit 8 had the narrowest range (0.5%–2.9%).

Use of SRNs and Patient Outcomes

In Table 2, we present the results of multilevel modeling on the effects of SRN use on patient outcomes after adjusting for patient characteristics, availability of RRT (used only with the mortality outcome model), and nurse staffing and unit characteristics. All observed effects of SRN use on patient outcomes were non-significant at alpha 0.05. In Table 2, we also present estimates and significance tests of two random effect parameters, unit (random intercept) and residuals for each outcome variable. The estimates of the random intercept variance components were non-significant for all outcomes except for the pressure ulcer outcome. These results indicate that the models adequately account for unit level variance for these outcomes (except PUs). All residual tests were significant, indicating the occurrence of unexplained (error) variance remaining at the unit quarter level on patient outcomes after adjusting for both level-1 and level-2 predictors.


Use of SRNs varied greatly by unit in the study hospital over time. Some units used SRNs consistently over time, whereas other units used SRNs sporadically. There was also substantial variation in the level of SRN use at any given time across units. These variations indicate that some units have consistently more vacant positions compared to other units and some units had more vacant positions at a single time. These 2 patterns may be associated with different underlying mechanisms for nurse staffing vacancies and demonstrate the importance of longitudinal data at the unit level in examining the relationship between SRN use and patient outcomes. Understanding the different patterns of SRN use and the factors that account for these patterns may shed further light on the relationships among SRN use, nursing work environment, and patient outcomes.

Our results suggest that use of SRNs had no significant impact, positive or negative, on patient outcomes. Several explanations might be pertinent to this finding. First, our finding is consistent with those of an earlier study3. These are the only 2 studies in which investigators have controlled for NWE and other covariates while examining the relationship between SRN use and patient outcomes. Controlling for NWE is critical because this factor may be linked to both SRN use and patient outcomes, and thus serves to confound the relationship between the two. Although no study has been conducted to examine the relationship between use of SRNs and organizational characteristics in acute care settings, a study conducted in a nursing home showed that use of agency nurses was associated with characteristics of poorer quality facilities28. Second, the current study examined a specific type of SRNs who were hired from several preferred staffing agencies and had 3 month renewable contracts. Experienced SRNs who had worked in similar hospital settings were preferred for hiring, and each SRN was provided a 3-day unit-level orientation. During the study period, the hospital did not use internal float nurses, so there was no contamination effect of float nurses on patient outcomes. The homogeneous sample of SRNs used in this study strengthened the internal validity of the findings and created an opportunity to distinguish and explore the relationship between this type of SRN and patient outcomes. This type of SRN might be different from per-diem SRNs because they have multiple months to adapt to the units. Therefore the effect of SRNs on patient outcomes might be different between per-diem SRNs and SRNs with multiple contracts. Future research is needed to understand the relationship between different types of SRNs and patient outcomes. Third, while our study findings might not generalize to other hospitals with different characteristics in different settings, our institution-based results are nonetheless informative, as Needleman et al. have recently demonstrated that a single site study can provide valid findings that are consistent with those of the multisite studies, due to the robust design, proper model specification and statistical controls, and unit level measures afforded by a single site study21. Future research with multisite study design is needed to replicate the findings.

In summary, the results of this study did not indicate a significant association between SRN use and patient outcomes. This finding has relevance for future nursing workforce development and policy. Although current levels of SRN use might be relatively low due to the economic downturn, future hospital use of SRNs is expected to rise with projected future nursing shortages. Therefore, the issue of the impact of SRN use on hospital quality of care will be more significant in the coming years. Future studies are needed to further explore the role of organizational and unit characteristics on SRN use and the relationship between the use of different types of SRNs on patient outcomes across a variety of hospital settings.

Supplementary Material



The authors gratefully thank and acknowledge Drs. Madeline Schmitt and James McMahon, University of Rochester School of Nursing, for their careful review and valuable comments on the manuscript.


This study was supported by the Robert Wood Johnson Foundation – Interdisciplinary Nursing Quality Research Initiative #62576. Dr. Noyes was also supported in part by the University of Rochester CTSA (UL1 RR024160) from the National Center for Research Resources (NCRR), a component of the National Institutes of Health (NIH) and the NIH Roadmap for Medical Research.

Contributor Information

Ying Xue, Associate Professor of Nursing, University of Rochester School of Nursing, Rochester, NY.

Linda H. Aiken, The Claire M. Fagin Leadership Professor of Nursing, Professor of Sociology, and Director of the Center for Health Outcomes and Policy Research, University of Pennsylvania, Philadelphia, PA.

Deborah A. Freund, President and University Professor, Claremont Graduate University, Claremont, CA.

Katia Noyes, Associate Professor and Chief, Division of Health Policy and Outcomes Research, Department of Preventive Medicine and Community Health, University of Rochester Medical Center, Rochester, NY.


1. First Consulting Group. The healthcare workforce shortage and its implications for America’s hospitals. The Association of American Medical Colleges, The Federation of American Hospitals, & The National Association of Public Hospitals and Health Systems; 2001. (Report commissioned by The American Hospital Association).
2. May JH, Bazzoli GJ, Gerland AM. Hospitals’ responses to nurse staffing shortages. Health Aff (Millwood) 2006;25(4):W316–323. [PubMed]
3. Aiken LH, Xue Y, Clarke SP, et al. Supplemental nurse staffing in hospitals and quality of care. J Nurs Adm. 2007;37(7–8):335–342. [PMC free article] [PubMed]
4. Alonso-Echanove J, Edwards JR, Richards MJ, et al. Effect of nurse staffing and antimicrobial-impregnated central venous catheters on the risk for bloodstream infections in intensive care units. Infect Control Hosp Epidemiol. 2003;24(12):916–925. [PubMed]
5. Estabrooks CA, Midodzi WK, Cummings GG, et al. The impact of hospital nursing characteristics on 30-day mortality. Nurs Res. 2005;54(2):74–84. [PubMed]
6. Robert J, Fridkin SK, Blumberg HM, et al. The influence of the composition of the nursing staff on primary bloodstream infection rates in a surgical intensive care unit. Infect Control Hosp Epidemiol. 2000;21(1):12–17. [PubMed]
7. Roseman C, Booker JM. Workload and environmental factors in hospital medication errors. Nurs Res. 1995;44(4):226–230. [PubMed]
8. Aiken LH, Clarke SP, Sloane DM, et al. Effects of hospital care environment on patient mortality and nurse outcomes. J Nurs Adm. 2008;38(5):223–229. [PMC free article] [PubMed]
9. Kramer M, Maguire P, Brewer BB. Clinical nurses in Magnet hospitals confirm productive, healthy unit work environments. J Nurs Manag. 2011;19(1):5–17. [PubMed]
10. McCusker J, Dendukuri N, Cardinal L, et al. Nursing work environment and quality of care: differences between units at the same hospital. Int J Health Care Qual Assur Inc Leadersh Health Serv. 2004;17(6):313–322. [PubMed]
11. American Nurses Association. Nursing-sensitive quality indicators for acute care settings and ANA’s safety & quality initiative 1999. Available at Accessed June 26, 2011.
12. NQF. National voluntary consensus standards for nursing-sensitive care: An initial performance measure set. Washington, DC: 2004.
13. Bryk AS, Raudenbush SW. Hierarchical linear models: applications and data analysis methods. Newbury Park, California: SAGE publications; 1992.
14. Milch CE, Salem DN, Pauker SG, et al. Voluntary electronic reporting of medical errors and adverse events. An analysis of 92,547 reports from 26 acute care hospitals. J Gen Intern Med. 2006;21(2):165–170. [PMC free article] [PubMed]
15. Press Ganey Associate Corporate. Press Ganey patient satisfaction. Available at Accessed March 22, 2010.
16. Bae SH, Mark B, Fried B. Use of temporary nurses and nurse and patient safety outcome in acute care hospital units. Health Care Manage Rev. 2010;35(3):333–344. [PubMed]
17. Kring DL. Reliability and validity of the Braden scale for predicting pressure ulcer risk. J Wound Ostomy Continence Nurs. 2007;34(4):399–406. [PubMed]
18. Deyo RA, Cherkin DC, Ciol MA. Adapting a clinical comorbidity index for use with ICD-9-CM administrative databases. J Clin Epidemiol. 1992;45(6):613–619. [PubMed]
19. Hillman K, DeVita M, Bellomo R, et al. Meta-analysis for rapid response teams. Arch Intern Med. 2010;170(11):996–997. [PubMed]
20. Kane R, Shamliyan T, Mueller C, et al. Nursing staffing and quality of patient care. Agency for Healthcare Research and Quality; Rockville, MD: 2007.
21. Needleman J, Buerhaus P, Pankratz VS, et al. Nurse staffing and inpatient hospital mortality. N Engl J Med. 2011;364(11):1037–1045. [PubMed]
22. Taunton RL, Bott MJ, Koehn ML, et al. The NDNQI-Adapted Index of work satisfaction. J Nurs Meas. 2004;12(2):101–122. [PubMed]
23. Lake ET. The nursing practice environment: measurement and evidence. Med Care Res Rev. 2007;64(2 Suppl):104S–122S. [PubMed]
24. Manojlovich M. Linking the practice environment to nurses’ job satisfaction through nurse-physician communication. J Nurs Scholarsh. 2005;37(4):367–373. [PubMed]
25. Snijders TAB, Bosker RJ. An introduction to basic and advanced multilevel modeling. Thousand Oaks, California: SAGE Publications; 1999.
26. Carpenter J, Goldstein H, Rasbash J. A novel boostrap procedure for assessing the relationship between class size and achievement. Applied Statistics. 2003;52(part 4):431–443.
27. Benjamini Y, Hochberg Y. Controlling the false discovery rate: A practical and powerful approach to multiple testing. Journal of the Royal Statistical Society Series B. 1995;57:289–300.
28. Bourbonniere M, Feng Z, Intrator O, et al. The use of contract licensed nursing staff in U.S. nursing homes. Med Care Res Rev. 2006;63(1):88–109. [PubMed]