|Home | About | Journals | Submit | Contact Us | Français|
Prior studies have highlighted wide variation in EMS workplace safety culture across agencies. We sought to determine the association between EMS workplace safety culture scores and patient or provider safety outcomes.
We administered a cross-sectional survey to EMS workers affiliated with a convenience sample of agencies. We recruited these agencies from a national EMS management organization. We used the EMS Safety Attitudes Questionnaire (EMS-SAQ) to measure workplace safety culture and the EMS Safety Inventory (EMS-SI), a tool developed to capture self-reported safety outcomes from EMS workers. The EMS-SAQ provides reliable and valid measures of six domains: safety climate, teamwork climate, perceptions of management, perceptions of working conditions, stress recognition, and job satisfaction. A panel of medical directors, paramedics, and occupational epidemiologists developed the EMS-SI to measure self-reported injury, medical errors and adverse events, and safety-compromising behaviors. We used hierarchical linear models to evaluate the association between EMS-SAQ scores and EMS-SI safety outcome measures.
Sixteen percent of all respondents reported experiencing an injury in the past 3 months, four of every 10 respondents reported an error or adverse event (AE), and 90% reported safety-compromising behaviors. Respondents reporting injury scored lower on 5 of the 6 domains of safety culture. Respondents reporting an error or AE scored lower for 4 of the 6 domains, while respondents reporting safety-compromising behavior had lower safety culture scores for 5 of 6 domains.
Individual EMS worker perceptions of workplace safety culture are associated with composite measures of patient and provider safety outcomes. This study is preliminary evidence of the association between safety culture and patient or provider safety outcomes.
Safety culture refers to the shared perceptions or attitudes of a work group towards safety.1 Leading authorities support regular safety culture assessments as part of healthcare organizational quality assurance.2 Prior studies of the in-hospital setting have linked safety culture scores to negative patient outcomes (more frequent medication errors, increased length of stay), provider outcomes (increased injury), and composite measures of safety outcomes.3–6 In high-risk occupations outside of healthcare, safety culture scores have been linked to occupational injuries, accidents, and safety audit measures.7–9
There is reason to believe that workplace safety culture impacts clinical and operational practices in Emergency Medical Services (EMS). Using structured instruments, prior studies have identified wide variation in EMS workplace safety culture.10,11 For example, air medical agencies scored higher than ground EMS agencies across all domains of safety culture.11 Lower safety culture scores were associated with higher annual call volume.11 However, no studies have associated workplace safety culture with patient or provider adverse events.
The objective of this study was to determine the association between EMS workplace safety culture and patient or provider safety outcomes.
We used a cross-sectional study design of a convenience sample of EMS workers. As an initial step, we sought to secure study participation from EMS agency level leaders prior to surveying individual EMS workers. We approached the National EMS Management Association (NEMSMA) and requested use of their member email Listserv to recruit EMS agency leaders. We distributed a standard letter to the NEMSMA email Listserv three times over an eight-week recruitment period. The NEMSMA is an all-inclusive professional association of 2,253 EMS leaders, managers, and administrators committed to best practices in EMS. A full description of NEMSMA member characteristics is not possible due to a lack of data maintained by the association. Twenty-seven agencies responded to our letter and completed the enrollment procedure. Most of these 27 agencies were located in the Northeast and Midwest U.S. Census regions (n=20, 74%), self-described third-service models (n=17, 63%), were a private non-profit ownership delivery model (n=16, 59%), and employed between 21 and 100 employees (n=20, 74%). The University of Pittsburgh Institutional Review Board approved this study.
We measured EMS safety using the EMS Safety Attitudes Questionnaire (EMS-SAQ). The EMS-SAQ is a 60-item survey that collects information on respondents’ opinions regarding perceptions of Safety Culture of an organization.10,11 The EMS-SAQ includes thirty core questions that measure six domains of Safety Culture: Safety Climate, Teamwork Climate, Perceptions of Management, Stress Recognition, Perceptions of Working Conditions, and Job Satisfaction. Responses to EMS-SAQ items are captured on a 5-point likert scale and weighted from 0 to 100 (Disagree Strongly=0, Disagree Slightly=25, Neutral=50, Agree Slightly=75, Agree Strongly=100).10–12 Standard scoring for the EMS-SAQ includes calculating a domain mean score and percentage positive of responses.10–12 We originally developed the EMS-SAQ by adapting the previously validated Intensive Care Unit Safety Attitudes Questionnaire (the ICU-SAQ).10–12 Prior studies involving both the EMS-SAQ and ICU-SAQ have demonstrated the tools to have positive psychometric properties of reliability and construct validity.
There are few tools for identifying EMS adverse events. Uniform criteria for identifying and rating safety outcomes in EMS do not exist, while standards for commonly held EMS data sources (i.e. the EMS patient care report) remain under development.13 While frequently used for safety research, electronic patient care reports are often incomplete and inaccurate.14–16 Self-reporting is a widely used and cost-efficient alternative to other forms of adverse event detection.17,18
We identified EMS provider and patient safety events using the EMS Safety Inventory (EMS-SI). The EMS-SI is a 44-item survey developed by a panel of EMS medical directors, EMTs and paramedics, and epidemiologists to identify events or behaviors with harm or risk of harm. We developed the EMS-SI using a modified Delphi process. After identifying candidate items from a literature review, a panel of experts [medical director physicians, prehospital providers, prehospital educators, and epidemiologists] identified items pertinent to provider or patient safety. We instructed the panel to use the AHRQ Patient Safety Indicator tool as a guide to classify EMS-SI items into one of three composite safety outcome measures: 1) provider injury (2-items; -e.g. “I was injured during a shift”), 2) patient care error or adverse event (25-items; e.g., “I accidently dislodged an ET tube”), and safety-compromising behaviors (17-items; e.g.“I have greatly exceeded the speed limit while responding lights and sirens”) (Appendix A). The time period of reference for the outcome measures was the previous three months.
We used two nominal, 7-option, categorical scales to elicit responses to EMS-SI items. Response options for each scale are detailed in Appendix A. Our expert panel considered “Probably Yes” and “Definitely Yes” as indicators of a negative safety outcome (aka: an affirmative response). The second nominal scale was used on a subset of 19-items assigned to the error or AE composite outcome measure. The expert panel considered “Ran Out of Time,” “Forgot to Perform,” and “Did Not Think it was Necessary” as indicators of a negative safety outcome.
Agency contacts provided key information on eligible respondents prior to survey dissemination. Key information included email address, certification level (EMT-Basic or paramedic), full-time status (Full-Time, Part-Time, or Volunteer), years of agency experience (e.g. 5 years), and percentage of total work devoted to clinical duties for all EMTs and paramedics (e.g. 75%). Agency contacts were instructed to consider all employees who worked clinically for the agency to be eligible, and for administrative personnel and non-clinical workers to be ineligible.
From January to June 2010 agency contacts distributed the survey to eligible EMTs and paramedics using our survey system. The system sent a standard email with a secure link to the survey. This email included an opt-out option (a secure link), that when clicked, documented the individual’s email address as opting out of all future emails from the survey system. We designed the survey system to provide agency contacts the ability to click a button that would resend the survey email and links to the survey and opt-out option to non-respondents. These emails were not sent to individuals that previously selected to opt out. The survey system stored all survey data on a secure server and assigned each completed survey a randomly generated survey ID number and unique agency ID number.
We used a respondent level analysis and calculated frequencies, percentages, means, and corresponding 95% confidence intervals for the EMS-SAQ and EMS-SI composite safety measures. We used Rao-Scott Chi-Square tests and Analysis of Variance (ANOVA) to examine associations between respondent characteristics, EMS-SAQ scores, and EMS-SI composite safety measures. We used hierarchical linear models to evaluate the association between EMS-SAQ domain scores and composite safety outcome measures, while accounting for within-agency clustering of respondents. We report the domain mean score and corresponding 95% confidence intervals for each EMS-SAQ domain stratified by the EMS-SI composite safety measure.
Twenty-seven agencies participated in the study. Six agencies failed to fulfill the study protocol within the six-month study period (completed the demographic information but did not send surveys to agency personnel), leaving responses from twenty-one agencies for analysis. We received 416 completed surveys (mean agency response rate of 42.1%; range 5.3% to 81.3%).
Three-quarters of participating agencies were located in either the Midwest (38.1%) or the Northeast (42.9%) census regions. Half (52%) operated as a 3rd service model and reported funding from local municipalities as a public service affiliate (Table 1). The private, non-profit financial model was the most common type of ownership (71.4%). Half of participating agencies employed between 51 and 100 personnel (Table 1).
Most respondents were white (90.6%), male (69.2%), between 26–35 years old (36.1%), and reported some college education (44.2%). Half were certified at the EMT-Paramedic (paramedic) level (51.2%; Table 2). Sixty percent of respondents reported ≥10 years of total EMS experience. Nearly half reported >5 years experienced at their current agency where employed (47.4%) (Table 2).
The aggregated SAQ domain mean scores were: Safety Climate (72.2; 67.0–77.5), Teamwork Climate (70.1; 63.9–76.2), Perceptions of Management (69.6; 62.9–76.2), Working Conditions (66.0; 58.5–73.6), Stress Recognition (54.3; 51.9–56.7), and Job Satisfaction (75.8; 69.7–81.9; Table 3).
One-sixth of all respondents reported having experienced an injury in previous three months (16%; Table 2). The reporting of injury was associated with provider age category (p<0.05). The mean domain scores for safety climate, teamwork climate, perceptions of management, working conditions, and job satisfaction were lower among respondents reporting injury than those who did not (p<0.05; Table 3).
Four out of every 10 respondents recorded an affirmative response for EMS-SI items measuring error or AE (42%; Table 2). Reporting of an error or AE was associated with provider age category and certification level (p<0.05). The mean domain scores for safety climate, teamwork climate, perceptions of management, and working conditions were lower among respondents with an affirmative response for an error or AE than domain mean scores for respondents without an error or AE (p<0.05; Table 3). The mean score for stress recognition was 4 points higher among respondents indicating an error or AE than among their respective reference group (P<0.05).
Nine of every ten respondents reported safety-compromising behaviors (89%; Table 2). The reporting of these behaviors was associated with provider age category and employment status (p<0.05). The mean domain scores for safety climate, teamwork climate, perceptions of management, working conditions, and job satisfaction were lower among respondents indicating safety-compromising behaviors than among respondents who denied such behaviors (P<0.05; Table 3).
This study provides preliminary evidence of an association between safety culture scores and safety outcomes in the EMS setting. In this study sample, we identified strong associations between five of the six domains of EMS workplace safety culture and the three composite measures of patient and provider safety outcomes. If independently validated, our findings have profound implications for EMS.
Prior studies have observed similar associations between workplace safety culture and healthcare outcomes. For example, a multicenter study of intensive care units linked higher rates of error to lower safety culture scores.4 Another study of 91 hospitals linked lower safety climate scores to a higher annual rate of a negative patient safety indicators adopted by the Agency for Healthcare Research and Quality.3
Although our research does not determine causality, it is an important finding because it suggests that EMS-SAQ safety culture survey scores can be used as a proxy measure for adverse event rates in individual EMS organizations.
We determined that the frequency of reported safety outcomes by paramedics exceeded the frequency of events reported by EMTs. There are several possible explanations for this difference. Paramedics provide a broader range of clinical care and interventions, including invasive procedures and medication administration, which would seem to create opportunity for more frequent negative outcomes. These individuals may be more apt to engage in risk taking behaviors.19 It is also possible that as a group paramedics are more comfortable reporting errors and adverse events, which would create a reporting bias. A substantial number of EMS-SI items query respondents on behaviors, actions, and procedures that are potentially more common among paramedics in most regions (i.e. “I did not perform a 12-lead ECG on a patient with chest pain”).
Our approach to identifying outcomes was conservative. We classified a respondent’s ‘affirmative’ response to any one item of a composite outcome towards the count of that outcome. Future revisions to the EMS-SI tool should encompass level-specific surveys. Employing a score of harm or severity to EMS-SI items may help investigators and EMS officials separate the severe from minor, further improving outcome identification.
A large body of recent safety research has emphasized the importance of systems and organizational factors as predictors of poor or positive safety outcomes.20–22 James Reason’s Swiss Cheese model of an accident is a widely recognized illustration of this concept.23 The climate or culture of an organization may be viewed as a slice of cheese positioned furthest away from the negative event/accident. Several studies have established a relationship between safety culture scores and worker injury or patient safety outcomes. There are numerous threats to EMS worker and patient safety. For example, patients are often violent, providers often deviate from protocol, partnerships are inconsistent and threaten teamwork, and injury rates among EMS workers exceed that of the general working public.24–32 The magnitude of each threat is poorly understood by the EMS industry due to a lack of adequate and standardized documentation of patient and provider safety events.33,34 In this study, we report an association between EMS workplace safety culture scores and safety outcome measures of injury, error, and adverse events. Our findings are early evidence that the EMS-SAQ is a valid indicator of EMS agency safety conditions. Lack of a statistically significant difference in safety outcomes and EMS-SAQ domain scores across demographic stratums is a worthy line of inquiry that deserves further exploration. These findings, while preliminary, suggest that the EMS work environment is associated with safety of patients and providers.
Our mean agency response rate (42.1%) is similar to previous studies of healthcare workers and safety culture studies, including prior studies of EMS agencies.35–40 Our analyses focused on the association between EMS-SAQ domain scores and EMS-SI composite outcomes at the individual EMS worker level. We did not analyze and interpret our findings from the agency perspective with the intent of comparing findings across agencies. However, we recognize that inclusion of data from low response agencies may introduce concern for the external validity of our data. One concern may be that EMS workers from agencies with a low response rates may not represent workplace safety culture in their agency as a whole and possibly provide an over representation of negative safety outcomes statistics. We examined our data from the agency level perspective and determined that domain mean scores for five of six EMS-SAQ domains were lower among respondents from agencies with low response rates (<60%) than agencies with higher response rates (≥60%). We determined that EMS workers from agencies with ≥60% agency level response rate had higher mean domain scores than agencies with <60% response rate for all domains except stress recognition. These differences are as follows: safety climate −12.9 (p=0.003), teamwork climate −12.0 (p=0.019), perceptions of management −16.8 (p=0.008), working conditions −19 (p=0.002), and job satisfaction −12.8 (p=0.011). We then determined that the cumulative frequency of EMS-SI composite outcomes findings were not statistically different when stratified by low versus high agency response rate: Injury =14.4% in high response agencies versus 19.2% in low response (p=0.5619); Error and AE = 40.0% in high response versus 41.7% in low response (p=0.9341); and Safety Compromising Behaviors = 85.6% in high response versus 88.6% in low response (p=0.3603).
Our findings may have limited generalizability to EMS workers from certain types of EMS delivery models (i.e. fire-based models). Part-time employees and volunteers are more common among non-respondents suggesting possible limited generalizability to these EMS workers. Perceptions of one or more domain of EMS-SAQ may be positively or negatively impacted if more part-time individuals responded. Additional research focused on part-time EMS workers would address this concern and shed light on the magnitude of possible differences in findings across employment status. Table 4 highlights observable similarities between our study sample and samples of workers from other multi-site studies. We believe these similarities provide an important frame of reference when interpreting our findings.
The EMS-SI tool provides insights into EMS global safety – not individual safety events. We chose this method because event-level detection is difficult, intensive, expensive, and yields information on only select events. Since our objective was to gauge overall safety rather than specific processes of care, the use of a composite measure of safety was appropriate. To ensure a robust and face-valid measure of safety outcomes, we used a modified Delphi consensus approach with a panel of medical directors, epidemiologists, and EMS workers to develop a unique self-reporting tool.41,42 The response options attached to each EMS-SI item were developed by our panel and designated appropriate. A panel composed of different individuals may have rendered a different tool, options for item response, and ultimately lead to different findings in this study. The EMS-SI tool is unique but comparable to a patient safety indicator tool developed by AHRQ and tested in prior large scale safety culture studies.3 We continue to edit and improve the EMS-SI.
The self-report nature of our safety outcome measures is a strength and weakness. We adopted a three-month period of recall in recognition that occupational epidemiologists consider the accuracy of recall to diminish beyond two months.43,44 We acknowledge that prior research has documented EMTs under report medical errors and adverse events by an estimated 4%.27,33,34 There is additional evidence that between 11% and 32% of occupational injuries and accidents are not reported.7,45 Under reporting may be attributed to an unwillingness to report, particularly in agencies where a poor safety culture creates a fear of retribution.46 Many EMS personnel may also fail to realize that an event has occurred, evidenced by studies on the procedure of synchronized cardioversion.47
We used a cross-sectional study design, which cannot determine causality or determine if poor safety culture scores preceded or were the result of negative safety outcomes. We believe our findings do not highlight the true strength of association between safety culture and safety outcomes. Our findings provide preliminary evidence that in one subset of the EMS population, safety culture is an important “meter of safety.” What is or is not a clinically meaningful difference in safety culture scores across the injured versus uninjured, for example, must be addressed. Findings from the current study provide the data required to design future research whereby investigators may wish to test hypotheses and detect clinically meaningful differences in safety climate scores across stratums of safety outcomes.
In this study, we detected associations between individual EMS worker perceptions of workplace safety culture scores and composite measures of patient and provider safety outcomes.