|Home | About | Journals | Submit | Contact Us | Français|
Workplace attitude, beliefs and culture may impact the safety of patient care. This study characterized perceptions of safety culture in a nationwide sample of Emergency Medical Services (EMS) agencies.
We conducted a cross-sectional survey involving 61 Advanced Life Support EMS agencies in North America. We administered a modified version of the Safety Attitudes Questionnaire (SAQ), a survey instrument measuring dimensions of workplace safety culture (Safety Climate, Teamwork Climate, Perceptions of Management, Job Satisfaction, Working Conditions, and Stress Recognition). We included full-time and part-time paramedics and Emergency Medical Technicians. We determined the variation in safety culture scores across EMS agencies. Using Hierarchical Linear Models (HLM), we determined associations between safety culture scores and individual and EMS agency characteristics.
We received 1,715 completed surveys from 61 EMS agencies (mean agency response rate 47%; 95% CI 10%, 83%). There was wide variation in safety culture scores across EMS agencies [mean (min, max)]: Safety Climate 74.5 (Min 49.9, Max 89.7), Teamwork Climate 71.2 (Min 45.1, Max 90.1), Perceptions of Management 67.2 (Min 31.1, Max 92.2), Job Satisfaction 75.4 (Min 47.5, Max 93.8), Working Conditions 66.9 (Min 36.6, Max 91.4), Stress Recognition 55.1 (Min 31.3, Max 70.6). Air medical EMS agencies tended to score higher across all safety culture domains. Lower safety culture scores were associated with increased annual patient contacts. Safety climate domain scores were not associated with other individual or EMS agency characteristics.
In this sample, workplace safety culture varies between EMS agencies.
Organizational safety culture refers to the collective beliefs and perceptions of workers regarding the organization and safety of their workplace operations.1 The Joint Commission, Agency for Healthcare Research and Quality, and Institute for Healthcare Improvement have all recommend frequent evaluations of organizational safety culture.2-4 The National EMS Advisory Council called upon the EMS industry to develop a “culture of safety.”5 Research in high reliability industries such as nuclear power and aviation have linked organizational safety culture to accidents, safety audit scores, and safety behavior.1, 6, 7 Previous research of in-hospital environments link safety culture scores to patient outcomes.8
Preventable adverse events occur in 1 of every 1,000 air medical EMS transports.9 A recent study has identified adverse events in ground EMS as well.10 Prehospital airway management errors are common and potentially harmful.11-13 Other studies and reviews highlight ambulance crashes, patient mishandling, malfunctioning equipment, medical mismanagement and protocol deviations.14-17 EMS personnel often report feeling stress and burnout.18-21 Other studies suggest that many Emergency Medical Technicians (EMTs) are concerned about the accuracy of care decisions, they suffer from poor sleep quality, high fatigue, and have a questionable commitment to their job.22-24 Both sleep and fatigue have been linked to medical error and performance in physicians and nurses.25-29 These observations suggest that EMS workplace culture may influence patient safety.
Prior studies have characterized organizational safety culture in the hospital inpatient setting, Intensive Care Units (ICUs), nursing wards, ambulatory care, and in skilled nursing facilities.30-33 However, there have been no descriptions of the safety culture in EMS. In this study we sought to characterize variation in workplace safety culture in EMS and test the psychometric properties of our survey tool.
This study was approved by the University of Pittsburgh Institutional Review Board. We performed a cross-sectional survey of North American EMS agencies.
We enrolled a convenience sample of 62 EMS agencies from the United States and Canada. We selected only agencies that provided Advanced Life Support Care. While we did not utilize a formal sampling scheme, we tried to include agencies from a range of practice and geographic settings. Currently, there are no reliable and valid lists of all EMS agencies in the U.S. Our strategy for recruitment included advertising on a single web site and circulating a study flyer on popular EMS leadership email Listservs.
We measured organizational safety culture using a structured 60-question survey instrument, the EMS Safety Attitudes Questionnaire (EMS-SAQ), which has been previously described.34 We developed the EMS-SAQ by modifying the Intensive Care Unit Safety Attitudes Questionnaire (ICU-SAQ); a widely used and validated survey instrument characterizing workplace safety culture in hospital critical care units.31, 35 Historically, the ICU-SAQ and related versions of the SAQ were based upon the Flight Management Attitudes Questionnaire, which assessed airline cockpit safety culture.30 The SAQ has been adapted for and validated in a range of medical settings such as ambulatory care, the operating room, ICU, and skilled nursing facilities.31-33, 36 We chose to modify the ICU-SAQ over other safety culture tools because it is widely used across different health care settings.37
The ICU-SAQ used 30 core questions to characterize 6 safety domains; 1) safety climate [7 items], 2) job satisfaction [5 items], 3) perceptions of management [4 items], 4) teamwork climate [6 items], 5) working conditions [4 items], and 6) stress recognition [4 items]. We retained the same domains in the EMS-SAQ. We modified the wording to ensure consistency with EMS practice and convention. For example, we changed “In the ICU, it is difficult to discuss mistakes.” to “At this EMS agency, it is difficult to discuss mistakes.” Respondents provided answers to each question using a 5-point Likert scale (Strongly Agree to Strongly Disagree) [See Appendix].
We determined that our initial version of the EMS-SAQ had positive psychometric properties.34 Specifically, tests revealed positive reliability and instrument validity scores. While we updated the instrument for this study, we did not alter the Core questions of the survey.
Eligible respondents were identified by each EMS agency. An individual was eligible if he/she worked full-time, part-time, or was a volunteer paramedic, EMT, first responder, prehospital nurse, or EMS physician who worked an average of at least 1 EMS shift per week. We excluded all managerial, administrative, or clerical personnel.
We administered the survey using two modalities: 1) paper forms, and 2) internet-based survey. Each EMS agency selected the modality deemed most convenient for the EMS agency. We permitted EMS agencies to use both modalities. The paper version of the survey consisted of questions printed in “bubble-sheet” format. Coordinators at each EMS agency supervised survey distribution. Respondents returned completed surveys in a self-addressed postage-paid envelope. We used a commercial survey vendor (keysurvey.com) to provide the electronic version of the survey. Each EMS agency provided the email for eligible employees. The vendor emailed a secure survey link to each potential participant. Up to three reminders were sent to non-respondents.
Completion of the survey was voluntary. The paper survey did not contain individual identifiers, and the electronic version was de-identified by the vendor prior to analysis. We used a separate paper-based instrument completed by the agency contact to determine characteristics of each EMS agency.
The primary outcomes were the survey scores for each safety domain: 1) safety climate [7 items], 2) job satisfaction [5 items], 3) perceptions of management [4 items], 4) teamwork climate [6 items], 5) working conditions [4 items], and 6) stress recognition [4 items]. We calculated the score for each domain using the method prescribed by Sexton, et al.30 We first converted each Likert ranking to a point scale ranging from 0 to 100; Disagree Strongly=0, Disagree Slightly=25, Neutral=50, Agree Slightly=75, and Agree Strongly=100. We calculated the domain score by adding the individual response scores and dividing by the total number of items. For example, if a respondent answered Disagree Strongly, Neutral, Neutral, and Agree Slightly on the 4 items of Stress Recognition, the domain score would be 43.75.
Prior efforts also dichotomized the safety domain scores to “positive” (domain score ≥75) and “non-positive” (domain score <75) responses.30 For example, if a respondent’s domain score for teamwork climate was 43.75, the respondent’s responses were classified as “non-positive.” To count as positive, a respondent would need to respond with an average response of Agree Slightly or higher. Following prior approaches by Sexton, et al., we examined these classifications as a proportion at the agency level and labeled it as the “percentage of positive responses” (PPR) [See Appendix].
EMS agency demographics were collected from agency contacts and included: agency type and geography, number of employees, number of annual patient contacts, agency affiliation, and percentage of patient contacts that were cardiac or trauma. EMS agencies designated their practice setting (i.e., rural ground, urban ground, air medical, or both ground and air medical). EMS agencies also reported their number of employees (i.e., 1-20 employees, 21-50, 51-100, or 101-400), number of annual patient contacts (i.e., ≤2,500, 2,501-5,000, 5,001-10,000, or >10,000), affiliation (i.e., hospital-based model, fire-based model, third service/government model, or private/free standing model), and the percentage of annual patient contacts that were cardiac arrests or trauma (i.e., ≤2% or >2%).
Individual characteristics included age category (i.e., 18-30, 31-40, 41-50, or >50), by total years of EMS experience (i.e., ≤5, 6-10, 11-15, 16-20, or >20), by total years at their current EMS agency (i.e., ≤5, 6-10, or >10), by position type (i.e., EMT-Basic, EMT-Intermediate, EMT-Paramedic, Prehospital Nurse, or Other), by employment status (i.e., career full-time, career part-time, or volunteer), by education (i.e., some high school, high school graduate or General Educational Development [GED], some college, college [e.g., Associate’s Degree or Bachelor’s], or college graduate level).
Like others before us,8, 30, 32, 38 we used Confirmatory Factor Analysis (CFA) to determine if the items administered to our targeted sample actually measured the hypothesized 6 domain survey model. We used Cronbach’s Alpha to examine the inter-correlations amongst the 6 EMS-SAQ domains. Scores range from 0-100 for each domain with higher values indicating that a set of items measure a single domain/construct.39, 40 Values lower than 0.70 raise questions about item wording and interpretation and whether or not the construct includes the appropriate number of items.39 We evaluated 3 standard measures of model fit to determine if our survey responses loaded onto a hypothesized 6 domain survey structure: the Root Mean Squared Error of Approximation (RMSEA), Bentler’s comparative fit index (CFI), and the Bentler and Bonett (1980) non-normed fit index (NNFI).40-42 An RMSEA less than 0.06, and CFI and NNFI greater than 0.9 are considered acceptable indices of instrument validity and model fit.40-42 The CFI and NNFI are less susceptible to sample size and considered compliments of the RMSEA.40-42
We calculated mean domain scores and PPR for each individual EMS agency, depicting the variation graphically and with descriptive statistics. To identify potential associations between domain scores and individual and EMS agency characteristics, we used hierarchical linear models, modeling EMS agency as a fixed effect. For each characteristic, we fit two models; a linear model for the raw domain score and a logit model for the percentage of positive responses. We examined associations between the 6 EMS-SAQ domains and agency and individual demographic variables. We performed all analyses using SAS v.9.1 (SAS Inc., Cary, North Carolina)
Of the 62 EMS agencies participating in the study, we excluded one agency due to low response rate (9.6%). The remaining 61 agencies were distributed across all 4 U.S. Census regions and included one EMS agency from Canada (Figure 1). Most were rural-ground EMS agencies (Table 1). Most agencies employed between 21 and 50 employees and were affiliated with a private/freestanding model. Approximately 43% of all EMS agencies had ≤2,500 patient contacts in 2007.
We received 1,715 completed surveys from 61 agencies. The mean survey response rate per EMS agency was 47% (95% CI 10%, 83%). Response rates varied slightly by method for survey completion (paper only [n=16]=52%, combination of electronic and paper [n=3]=49%, and electronic only [n=42]=45%). We excluded 120 surveys that were missing 2 or more demographic variables. The final analysis included 1,595 surveys.
In this sample of EMS personnel, the survey responses demonstrated good internal consistency (Cronbach’s alpha for safety climate, alpha=0.82; teamwork climate, alpha=0.83; perceptions of management, alpha=0.68; job satisfaction, alpha=0.8; working conditions, alpha=0.75, stress recognition, alpha=0.78). Instrument validity testing confirmed the presence of a 6 domain structure and good model fit properties: RMSEA= 0.04, CFI=0.97, NNFI=0.95.
Most respondents were male and EMT-Paramedic certified (Table 2). The most common stratums were 18-30 (27.4%) and 31-40 (37.3%). The most common stratum for total years of EMS experience was less than 5 years (28.5%). The most common stratum for total years of experience at the current EMS agency was less than 5 years (44.9%). Three quarters of respondents (77.6%) were career full-time employees and half (50.3%) had an associate’s or bachelor’s degree.
Agency mean domain scores varied across EMS agencies (Figures 2a-2f); Safety Climate 74.5 (95%CI 72.4, 76.6; Min 49.9, Max 89.7), Teamwork Climate 71.2 (95%CI 68.6, 73.7; Min 45.1, Max 90.1), Perceptions of Management 67.2 (95%CI 63.9, 70.5; Min 31.1, Max 92.2), Job Satisfaction 75.4 (95%CI 72.8, 78.0; Min 47.5, Max 93.8), Working Conditions 66.9 (95%CI 64.0, 69.7; Min 36.6, Max 91.4), Stress Recognition 55.1 (95%CI 52.9, 57.2; Min 31.3, Max 70.6).
The mean safety climate score for air-medical EMS agencies was greater than mean scores in private/free standing and fire based model agencies (Table 3). The mean safety climate score was also highest in agencies with fewer employees, agencies with lower annual patient contacts, and agencies with a higher proportion of acute patients.
The percentage of respondents with a positive perception (PPR) varied across EMS agencies for each domain: Safety Climate 58.6% (95%CI 52.7, 64.5; Min 0%, Max 100%), Teamwork Climate 52.8% (95%CI 46.5, 59.1; Min 0%, Max 100%), Perceptions of Management 43.8% (95%CI 36.7, 50.9; Min 0%, Max 90%), Job Satisfaction 63.4% (95%CI 57.5, 69.4; Min 0%, Max 100%), Working Conditions 46.3% (95%CI 41.4, 51.3; Min 0%, Max 100%), Stress Recognition 28.6% (95%CI 25.2, 32.0; Min 0%, Max 57.1%) (Figures 3a-3f).
The PPR for safety climate was highest among air-medical only agencies, but did not differ significantly across categories of model affiliation (Table 4). A lower proportion of EMS agency patient contacts was associated with increased PPR for safety climate. EMS agencies with greater than or equal to 2% of trauma and cardiac related patient contacts had a higher PPR than agencies with less than 2%. Notably, the PPR for stress recognition did not vary across any of the selected agency characteristics.
The mean safety climate score was lower for EMT-Paramedics than the mean score for all other position types (Table 5). The mean safety climate scores were highest among respondents between the ages of 41 and 50, highest among respondents with less experience in EMS, and highest among respondents with less experience at the current EMS agency of employment when compared to their respective referent groups.
The PPR for safety climate did not differ across most respondent demographic factors (Table 6). The PPR for safety climate was highest among Prehospital Nurses and other positions and was lowest among paramedics (p<0.0001; Table 6). The PPR for other domains of safety culture varied across some, but not all respondent demographic characteristics. Notably, the PPR for stress recognition varied across 1 of the 7 measured respondent demographics; Education. The PPR for stress recognition increased with categories of higher education.
This study proposes and tests a survey adapted for EMS from a previously validated safety culture survey. Safety culture assessments are now common practice across most healthcare organizations. These assessments serve multiple purposes, including setting safety benchmarks, targeting problem areas, program evaluation, and meeting regulatory requirements. In this study sample, we observed wide interagency variation in workplace safety culture. Scores at the lower of this variation raise the question: Is the patient’s safety more susceptible in these EMS agencies much more so than in agencies with higher EMS-SAQ scores? Conversely, do higher scores suggest that select EMS agencies hold a greater awareness of safety and practice accordingly?
Wide variation in workplace safety culture is not surprising given that the EMS work environment contains many threats to patient and provider safety. Suyama, et al. showed that in one urban environment injury rates associated with lost time at work were higher among paramedics and EMTs than fire and police.43 In a study of two urban EMS agencies, Maguire, et al. determined that the risk of injury among EMS personnel was 1.5 times higher than firefighters, 5.8 times higher than health services personnel, and 7 times higher than the national average reported by the U.S. Department of Labor.44 Other studies show that many EMS personnel often deviate from written protocols, fail to properly secure patient airways, experience high levels of stress and burnout, suffer from poor sleep quality and high fatigue, and have a questionable commitment to the profession.15, 19, 24, 45-48 When combined, these factors may surface as non-positive perceptions of worker safety culture.49
Prior safety culture studies have identified a pattern of variation in safety culture scores across settings. In an international study of safety culture, Sexton and colleagues’ identified wide variation in safety culture scores across 203 clinical units (i.e., operating rooms, ambulatory care settings, and ICUs).30 A statewide study of ICU safety culture in Michigan revealed wide variation in scores across ICUs in a single state.8 Positive perceptions of teamwork climate ranged from a low of 16% to 92%. In a study of 4 ICUs, Huang et al discovered significant variation in scores within a single institution, with positive perceptions of safety climate ranging from approximately 30% to 50% and positive perceptions of job satisfaction ranging from 20% to 70% across ICUs.31 In this context our observation of wide safety culture variation across EMS agencies is not surprising. Potential factors underlying culture variation include regional practice differences, varying economic resources, and different leadership structure and styles.
A common mechanism for error or adverse event classification and reporting does not exist. Measurement of adverse events and medical errors in EMS is difficult. Research by Hobgood, et al. suggests that there is limited reliability and accuracy in paramedics and EMTs self-report of errors.50 Threats to safety may be present absent actual errors or adverse events. While we believe that culture instruments could complement – but should not replace – direct AE measurement, culture is a potential contributor to poor safety.51 The overall utility of workplace safety culture instruments may lie in their ability to highlight safety conditions at individual EMS agencies. The EMS-SAQ offers a novel approach to patient safety, providing a barometer of safety attitudes rather than direct measures like errors or adverse events.
While our convenience sample may not represent all ALS level EMS agencies in Northern America, few sampling alternatives exist. A valid and reliable list of all EMS agencies in the U.S. does not exist. Sampling individual EMS personnel akin to the National Registry’s Longitudinal Emergency Medical Technician Attributes and Demographics Study (LEADS) is not appropriate for studies of workplace safety culture.
Approximately 32% of our study sample includes EMS agencies from Minnesota. Among these agencies, mean domain scores were slightly lower for 5 of the 6 domains compared to all other agencies. Safety climate (70.7 vs. 76.6; p=0.006), teamwork climate (66.6 vs. 73.7; p=0.005), stress recognition (51.0 vs. 57.4; p=0.003), perceptions of working conditions (61.7 vs. 69.7; p=0.005), perceptions of management (63.9 vs. 68.9; p=0.141), job satisfaction (71.8 vs. 77.4; p=0.041). Findings should not be interpreted as representative of EMS nationwide.
Response rates across EMS agencies were similar to other studies of EMS agencies and individual EMS workers.46, 52 Notably, we observed a slightly better average response rate in this study compared to other multi-site studies of safety culture.53-56 Individual agency response rates below 60% may reduce the accuracy of perceptions of safety.57 In our study, agencies with ≥60% (n=12) agency level response rates were not significantly different (p>0.05) than agencies with a response rate ≤59% (n=49) when comparing agency type and geography, number of employees, total patient contacts, or by percentage of patient contacts that were cardiac arrest or trauma. However, mean domain scores for perceptions of management and satisfaction were lower among agencies with a lower response rate than among agencies with a higher response rate (p<0.05).
Both respondent and agency factors may help explain differences in EMS-SAQ scores and thus represent residual confounding. Our study was not designed to identify a likely list of agency and respondent characteristics predictive of variations in safety culture scores. Rather, the primary purpose of our study was to characterize safety culture in the EMS setting using a reliable and valid measure of safety culture. Nonetheless, we conducted additional analyses to test for such differences which may be used to develop testable hypotheses in future research. We employed 12 hierarchical linear models with the 6 EMS-SAQ domain mean scores and 6 PPR proportions as outcomes and agency and individual variables as independent variables. We found few and potentially clinically insignificant differences in EMS-SAQ scores across agency and individual characteristics.
We did not examine the linkage between EMS-SAQ scores and adverse events or medical errors as in a recent studies of safety culture.8 Exploring this linkage in EMS would be methodologically difficult. Identifying medical errors and adverse events is a time-intensive exercise for which standards for identification and classification are limited.10
Workplace safety culture varies widely in this sample of EMS agencies. The EMS-SAQ can provide insights into prehospital safety.
Disclosure of funding
This study was supported by grants from the MedEvac Foundation International (www.fareonline.org) and Pittsburgh Emergency Medicine Foundation (PEMF www.pemf.net). During the execution of this study, Dr. Patterson was supported by the Emergency Medicine Patient Safety Foundation (EMPSF) and Society for Academic Emergency Medicine (SAEM) Patient Safety Research Fellowship. Dr. Wang was supported by Clinical Scientist Development Award K08-HS013628 from the Agency for Healthcare Research and Quality, Rockville, MD. Dr. Patterson is currently supported by a KL2 grant (KL2 RR024154) from the National Center for Research Resources (NCRR), a component of the National Institutes of Health (NIH), and NIH Roadmap for Medical Research. Dr. Fairbanks is supported by a Career Development Award from the NIBIB, K08-EB009090. The contents of this article are solely the responsibility of the authors and do not necessarily represent the official view of NCRR or NIH. Information on NCRR is available at http://www.ncrr.nih.gov/. Information on Re-engineering the Clinical Research Enterprise can be obtained from http://nihroadmap.nih.gov/clinicalresearch/overviewtranslational.asp.”
Questions of the Emergency Medical Service Safety Attitudes Questionnaire (EMS-SAQ)
(1=Disagree Strongly, 2=Disagree Slightly, 3=Neutral, 4=Agree Slightly, 5=Agree Strongly)
|a. Safety Climate Domain||(Questions: 3, 4, 8, 9, 14, 15, 22)|
|b. Teamwork Climate Domain||(Questions: 2, 17, 24, 27, 28, 29)|
|c. Stress Recognition Domain||(Questions: 18, 25, 26, 34)|
|d. Perceptions of Management Domain||(Questions: 7, 12, 13, 19)|
|e. Working Conditions Domain||(Questions: 5, 16, 32, 44)|
|f. Job Satisfaction Domain||(Questions: 1, 6, 11, 23, 31)|
The percentage of positive responses (PPR) is the proportion of respondents that have a positive perception of a domain.30
P. Daniel Patterson, University of Pittsburgh School of Medicine, Department of Emergency Medicine and Center for Emergency Medicine of Western Pennsylvania, Inc., Iroquois Building, 3600 Forbes Avenue, Suite 400A, Pittsburgh, PA 15261, Email: pattersond/at/upmc.edu.
David T. Huang, University of Pittsburgh School of Medicine, Department of Emergency Medicine and Center for Emergency Medicine of Western Pennsylvania, Inc., Iroquois Building, 3600 Forbes Avenue, Suite 400A, Pittsburgh, PA 15261; University of Pittsburgh School of Medicine, Department of Critical Care Medicine and Clinical Research and Systems Modeling of Acute Illness Laboratory [CRISMA], Email: huangdt/at/ccm.upmc.edu.
Rollin J. Fairbanks, University of Rochester School of Medicine, Department of Emergency Medicine, Email: terry.fairbanks/at/rochester.edu.
Scott Simeone, University of Pittsburgh School of Medicine, Department of Emergency Medicine and Center for Emergency Medicine of Western Pennsylvania, Inc., Iroquois Building, 3600 Forbes Avenue, Suite 400A, Pittsburgh, PA 15261, Email: simeones/at/upmc.edu.
Matthew Weaver, University of Pittsburgh School of Medicine, Department of Emergency Medicine and Center for Emergency Medicine of Western Pennsylvania, Inc., Iroquois Building, 3600 Forbes Avenue, Suite 400A, Pittsburgh, PA 15261, Email: weavermd/at/upmc.edu.
Henry E. Wang, University of Alabama at Birmingham School of Medicine, Department of Emergency Medicine, Email: hwang/at/uabmc.edu.