Background and Purpose. In this paper, we present a protocol for advanced psychometric assessments of surveys based on the Standards for Educational and Psychological Testing. We use the Alberta Context Tool (ACT) as an exemplar survey to which this protocol can be applied. Methods. Data mapping, acceptability, reliability, and validity are addressed. Acceptability is assessed with missing data frequencies and the time required to complete the survey. Reliability is assessed with internal consistency coefficients and information functions. A unitary approach to validity consisting of accumulating evidence based on instrument content, response processes, internal structure, and relations to other variables is taken. We also address assessing performance of survey data when aggregated to higher levels (e.g., nursing unit). Discussion. In this paper we present a protocol for advanced psychometric assessment of survey data using the Alberta Context Tool (ACT) as an exemplar survey; application of the protocol to the ACT survey is underway. Psychometric assessment of any survey is essential to obtaining reliable and valid research findings. This protocol can be adapted for use with any nursing survey.
Researchers strive to optimize data quality in order to ensure that study findings are valid and reliable. In this paper, we describe a data quality control program designed to maximize quality of survey data collected using computer-assisted personal interviews. The quality control program comprised three phases: (1) software development, (2) an interviewer quality control protocol, and (3) a data cleaning and processing protocol. To illustrate the value of the program, we assess its use in the Translating Research in Elder Care Study. We utilize data collected annually for two years from computer-assisted personal interviews with 3004 healthcare aides. Data quality was assessed using both survey and process data. Missing data and data errors were minimal. Mean and median values and standard deviations were within acceptable limits. Process data indicated that in only 3.4% and 4.0% of cases was the interviewer unable to conduct interviews in accordance with the details of the program. Interviewers' perceptions of interview quality also significantly improved between Years 1 and 2. While this data quality control program was demanding in terms of time and resources, we found that the benefits clearly outweighed the effort required to achieve high-quality data.
Although organizational context is central to evidence-based practice, underdeveloped measurement hindersitsassessment. The Alberta Context Tool, comprised of 59 items that tap10 modifiable contextual concepts, was developed to address this gap. The purpose of this study to examine the reliability and validity of scores obtained when the Alberta Context Tool is completed by professional nurses across different healthcare settings. Five separate studies (N = 2361 nurses across different care settings) comprised the study sample. Reliability and validity were assessed. Cronbach’s alpha exceeded 0.70 for9/10 Alberta Context Tool concepts. Item-total correlations exceeded acceptable standards for 56/59items. Confirmatory Factor Analysescoordinated acceptably with the Alberta Context Tool’s proposed latent structure. The mean values for each Alberta Context Tool concept increased from low to high levels of research utilization(as hypothesized) further supporting its validity. This study provides robust evidence forreliability and validity of scores obtained with the Alberta Context Tool when administered to professional nurses.
Older adults living in residential long-term care or nursing homes have increasingly complex needs, including more dementia than in the past, yet we know little about the unregulated workforce providing care. We surveyed 1,381 care aides in a representative sample of 30 urban nursing homes in the three Canadian Prairie provinces and report demographic, health and well-being, and work-related characteristics. Over 50 per cent of respondents were not born in Canada and did not speak English as their first language. They reported moderately high levels of burnout and a strong sense of their work’s worth. Few respondents reported attending educational sessions. This direct caregiver workforce is poorly understood, has limited training or standards for minimum education, and training varies widely across provinces. Workplace characteristics affecting care aides reflect factors that precipitate burnout in allied health professions, with implications for quality of care, staff health, and staff retention.
aging; care aides; long-term care; nursing homes; personal care workers; unregulated healthcare workforce; vieillissment; aides de soins; soins a longue durée; main d'oeuvre de services de santé non-règlementé
Poor communication among healthcare professionals is a pressing problem, contributing to widespread barriers to patient safety. The word “communication” means to share or make common. In the literature, two communication paradigms dominate: (1) communication as a transactional process responsible for information exchange, and (2) communication as a transformational process responsible for causing change. Implementation science has focused on information exchange attributes while largely ignoring transformational attributes of communication. In this paper, we debate the merits of encompassing both paradigms.
We conducted a two-staged literature review searching for the concept of communication in implementation science to understand how communication is conceptualized. Twenty-seven theories, models, or frameworks were identified; only Rogers’ Diffusion of Innovations theory provides a definition of communication and includes both communication paradigms. Most models (notable exceptions include Diffusion of Innovations, The Ottawa Model of Research Use, and Normalization Process Theory) describe communication as a transactional process. But thinking of communication solely as information transfer or exchange misrepresents reality. We recommend that implementation science theories (1) propose and test the concept of shared understanding when describing communication, (2) acknowledge that communication is multi-layered, identify at least a few layers, and posit how identified layers might affect the development of shared understanding, (3) acknowledge that communication occurs in a social context, providing a frame of reference for both individuals and groups, (4) acknowledge the unpredictability of communication (and healthcare processes in general), and (5) engage with and draw on work done by communication theorists.
Implementation science literature has conceptualized communication as a transactional process (when communication has been mentioned at all), thereby ignoring a key contributor to implementation intervention success. When conceptualized as a transformational process, the focus of communication moves to shared understanding and is grounded in human interactions and the way we go about constructing knowledge. Instead of hiding in plain sight, we suggest explicitly acknowledging the role that communication plays in our implementation efforts. By using both paradigms, we can investigate when communication facilitates implementation, when it does not, and how to improve it so that our implementation and clinical interventions are embraced by clinicians and patients alike.
Communication; Physician-nurse relations; Theoretical models; Knowledge translation
One of the greatest challenges in healthcare is how to best translate research evidence into clinical practice, which includes how to change health-care professionals’ behaviours. A commonly held view is that multifaceted interventions are more effective than single-component interventions. The purpose of this study was to conduct an overview of systematic reviews to evaluate the effectiveness of multifaceted interventions in comparison to single-component interventions in changing health-care professionals’ behaviour in clinical settings.
The Rx for Change database, which consists of quality-appraised systematic reviews of interventions to change health-care professional behaviour, was used to identify systematic reviews for the overview. Dual, independent screening and data extraction was conducted. Included reviews used three different approaches (of varying methodological robustness) to evaluate the effectiveness of multifaceted interventions: (1) effect size/dose-response statistical analyses, (2) direct (non-statistical) comparisons of multifaceted to single interventions and (3) indirect comparisons of multifaceted to single interventions.
Twenty-five reviews were included in the overview. Three reviews provided effect size/dose-response statistical analyses of the effectiveness of multifaceted interventions; no statistical evidence of a relationship between the number of intervention components and the effect size was found. Eight reviews reported direct (non-statistical) comparisons of multifaceted to single-component interventions; four of these reviews found multifaceted interventions to be generally effective compared to single interventions, while the remaining four reviews found that multifaceted interventions had either mixed effects or were generally ineffective compared to single interventions. Twenty-three reviews indirectly compared the effectiveness of multifaceted to single interventions; nine of which also reported either a statistical (dose-response) analysis (N = 2) or a non-statistical direct comparison (N = 7). The majority (N = 15) of reviews reporting indirect comparisons of multifaceted to single interventions showed similar effectiveness for multifaceted and single interventions when compared to controls. Of the remaining eight reviews, six found single interventions to be generally effective while multifaceted had mixed effectiveness.
This overview of systematic reviews offers no compelling evidence that multifaceted interventions are more effective than single-component interventions.
Electronic supplementary material
The online version of this article (doi:10.1186/s13012-014-0152-6) contains supplementary material, which is available to authorized users.
A shortage of transplantable organs is a global problem. There are two types of organ donation: living and deceased. Deceased organ donation can occur following neurological determination of death (NDD) or cardiocirculatory death. Donation after cardiocirculatory death (DCD) accounts for the largest increments in deceased organ donation worldwide. Variations in the use of DCD exist, however, within Canada and worldwide. Reasons for these discrepancies are largely unknown. The purpose of this study is to develop, implement, and evaluate a theory-based knowledge translation intervention to provide practical guidance about how to increase the numbers of DCD organ donors without reducing the numbers of standard NDD donors.
We will use a mixed method three-step approach. In step one, we will conduct semi-structured interviews, informed by the Theoretical Domains Framework, to identify and describe stakeholders’ beliefs and attitudes about DCD and their perceptions of the multi-level factors that influence DCD. We will identify: determinants of the evidence-practice gap; specific behavioural changes and/or process changes needed to increase DCD; specific group(s) of clinicians or organizations (e.g., provincial donor organizations) in need of behaviour change; and specific targets for interventions. In step two, using the principles of intervention mapping, we will develop a theory-based knowledge translation intervention that encompasses behavior change techniques to overcome the identified barriers and enhance the enablers to DCD. In step three, we will roll out the intervention in hospitals across the 10 Canadian provinces and evaluate its effectiveness using a multiple interrupted time series design.
We will adopt a behavioural approach to define and test novel, theory-based, and ethically-acceptable knowledge translation strategies to increase the numbers of available DCD organ donors in Canada. If successful, this study will ultimately lead to more transplantations, reducing patient morbidity and mortality at a population-level.
Nurses and allied health care professionals (physiotherapists, occupational therapists, speech and language pathologists, dietitians) form more than half of the clinical health care workforce and play a central role in health service delivery. There is a potential to improve the quality of health care if these professionals routinely use research evidence to guide their clinical practice. However, the use of research evidence remains unpredictable and inconsistent. Leadership is consistently described in implementation research as critical to enhancing research use by health care professionals. However, this important literature has not yet been synthesized and there is a lack of clarity on what constitutes effective leadership for research use, or what kinds of intervention effectively develop leadership for the purpose of enabling and enhancing research use in clinical practice. We propose to synthesize the evidence on leadership behaviours amongst front line and senior managers that are associated with research evidence by nurses and allied health care professionals, and then determine the effectiveness of interventions that promote these behaviours.
Using an integrated knowledge translation approach that supports a partnership between researchers and knowledge users throughout the research process, we will follow principles of knowledge synthesis using a systematic method to synthesize different types of evidence involving: searching the literature, study selection, data extraction and quality assessment, and analysis. A narrative synthesis will be conducted to explore relationships within and across studies and meta-analysis will be performed if sufficient homogeneity exists across studies employing experimental randomized control trial designs.
With the engagement of knowledge users in leadership and practice, we will synthesize the research from a broad range of disciplines to understand the key elements of leadership that supports and enables research use by health care practitioners, and how to develop leadership for the purpose of enhancing research use in clinical practice.
Dietetics,implementation science; Knowledge translation; Leadership; Management; Nursing; Occupational therapy; Physiotherapy; Research use; Speech and language pathology
To demonstrate the benefit of defining operational management units in nursing homes and computing quality indicators on these units as well as on the whole facility.
Calculation of adjusted Resident Assessment Instrument – Minimum Data Set 2.0 (RAI–MDS 2.0) quality indicators for: PRU05 (prevalence of residents with a stage 2–4 pressure ulcer), PAI0X (prevalence of residents with pain) and DRG01 (prevalence of residents receiving an antipsychotic with no diagnosis of psychosis), for quarterly assessments between 2007 and 2011 at unit and facility levels. Comparisons of these risk-adjusted quality indicators using statistical process control (control charts).
A representative sample of 30 urban nursing homes in the three Canadian Prairie Provinces.
Explicit decision rules were developed and tested to determine whether the control charts demonstrated improving, worsening, unchanging or unclassifiable trends over the time period. Unit and facility performance were compared.
In 48.9% of the units studied, unit control chart performance indicated different changes in quality over the reporting period than did the facility chart. Examples are provided to illustrate that these differences lead to quite different quality interventions.
Our results demonstrate the necessity of considering facility-level and unit-level measurement when calculating quality indicators derived from the RAI–MDS 2.0 data, and quite probably from any RAI measures.
Quality of Care; Performance Measurement; Unit; Clinical Microsystem; Nursing Homes
To study the association between organizational context and research utilization in German residential long term care (LTC), we translated three Canadian assessment instruments: the Alberta Context Tool (ACT), Estabrooks’ Kinds of Research Utilization (RU) items and the Conceptual Research Utilization Scale. Target groups for the tools were health care aides (HCAs), registered nurses (RNs), allied health professionals (AHPs), clinical specialists and care managers. Through a cognitive debriefing process, we assessed response processes validity–an initial stage of validity, necessary before more advanced validity assessment.
We included 39 participants (16 HCAs, 5 RNs, 7 AHPs, 5 specialists and 6 managers) from five residential LTC facilities. We created lists of questionnaire items containing problematic items plus items randomly selected from the pool of remaining items. After participants completed the questionnaires, we conducted individual semi-structured cognitive interviews using verbal probing. We asked participants to reflect on their answers for list items in detail. Participants’ answers were compared to concept maps defining the instrument concepts in detail. If at least two participants gave answers not matching concept map definitions, items were revised and re-tested with new target group participants.
Cognitive debriefings started with HCAs. Based on the first round, we modified 4 of 58 ACT items, 1 ACT item stem and all 8 items of the RU tools. All items were understood by participants after another two rounds. We included revised HCA ACT items in the questionnaires for the other provider groups. In the RU tools for the other provider groups, we used different wording than the HCA version, as was done in the original English instruments. Only one cognitive debriefing round was needed with each of the other provider groups.
Cognitive debriefing is essential to detect and respond to problematic instrument items, particularly when translating instruments for heterogeneous, less well educated provider groups such as HCAs. Cognitive debriefing is an important step in research tool development and a vital component of establishing response process validity evidence. Publishing cognitive debriefing results helps researchers to determine potentially critical elements of the translated tools and assists with interpreting scores.
Cognitive debriefing; Translation; Alberta context tool; Estabrooks’ kind of research utilization items; Conceptual research utilization scale; Organizational context; Research utilization; Residential long term care
Understanding the relationship between organizational context and research utilization is key to reducing the research-practice gap in health care. This is particularly true in the residential long term care (LTC) setting where relatively little work has examined the influence of context on research implementation. Reliable, valid measures and tools are a prerequisite for studying organizational context and research utilization. Few such tools exist in German. We thus translated three such tools (the Alberta Context Tool and two measures of research use) into German for use in German residential LTC. We point out challenges and strategies for their solution unique to German residential LTC, and demonstrate how resolving specific challenges in the translation of the health care aide instrument version streamlined the translation process of versions for registered nurses, allied health providers, practice specialists, and managers.
Our translation methods were based on best practices and included two independent forward translations, reconciliation of the forward translations, expert panel discussions, two independent back translations, reconciliation of the back translations, back translation review, and cognitive debriefing.
We categorized the challenges in this translation process into seven categories: (1) differing professional education of Canadian and German care providers, (2) risk that German translations would become grammatically complex, (3) wordings at risk of being misunderstood, (4) phrases/idioms non-existent in German, (5) lack of corresponding German words, (6) limited comprehensibility of corresponding German words, and (7) target persons’ unfamiliarity with activities detailed in survey items. Examples of each challenge are described with strategies that we used to manage the challenge.
Translating an existing instrument is complex and time-consuming, but a rigorous approach is necessary to obtain instrument equivalence. Essential components were (1) involvement of and co-operation with the instrument developers and (2) expert panel discussions, including both target group and content experts. Equivalent translated instruments help researchers from different cultures to find a common language and undertake comparative research. As acceptable psychometric properties are a prerequisite for that, we are currently carrying out a study with that focus.
Translation; Cultural adaptation; Alberta Context Tool; Estabrooks’ Kind of Research Utilization; Conceptual Research Use scale; Organizational context; Research utilization; Residential long term care
Organizational context is recognized as an important influence on the successful implementation of research by healthcare professionals. However, there is relatively little empirical evidence to support this widely held view.
The objective of this study was to identify dimensions of organizational context and individual (nurse) characteristics that influence pediatric nurses’ self-reported use of research. Data on research use, individual, and contextual variables were collected from registered nurses (N = 735) working on 32 medical, surgical and critical care units in eight Canadian pediatric hospitals using an online survey. We used Generalized Estimating Equation modeling to account for the correlated structure of the data and to identify which contextual dimensions and individual characteristics predict two kinds of self-reported research use: instrumental (direct) and conceptual (indirect).
Significant predictors of instrumental research use included: at the individual level; belief suspension-implement, research use in the past, and at the hospital unit (context) level; culture, and the proportion on nurses possessing a baccalaureate degree or higher. Significant predictors of conceptual research use included: at the individual nurse level; belief suspension-implement, problem solving ability, use of research in the past, and at the hospital unit (context) level; leadership, culture, evaluation, formal interactions, informal interactions, organizational slack-space, and unit specialty.
Hospitals, by focusing attention on modifiable elements of unit context may positively influence nurses’ reported use of research. This influence of context may extend to the adoption of best practices in general and other innovative or quality interventions.
There is emerging evidence that context is important for successful transfer of research knowledge into health care practice. The Alberta Context Tool (ACT) is a Canadian developed research-based instrument that assesses 10 modifiable concepts of organizational context considered important for health care professionals’ use of evidence. Swedish and Canadian health care have similarities in terms of organisational and professional aspects, suggesting that the ACT could be used for measuring context in Sweden. This paper reports on the translation of the ACT to Swedish and a testing of preliminary aspects of its validity, acceptability and reliability in Swedish elder care.
The ACT was translated into Swedish and back-translated into English before being pilot tested in ten elder care facilities for response processes validity, acceptability and reliability (Cronbach’s alpha). Subsequently, further modification was performed.
In the pilot test, the nurses found the questions easy to respond to (52%) and relevant (65%), yet the questions’ clarity were mainly considered ‘neither clear nor unclear’ (52%). Missing data varied between 0 (0%) and 19 (12%) per item, the most common being 1 missing case per item (15 items). Internal consistency (Cronbach’s Alpha > .70) was reached for 5 out of 8 contextual concepts. Translation and back translation identified 21 linguistic- and semantic related issues and 3 context related deviations, resolved by developers and translators.
Modifying an instrument is a detailed process, requiring time and consideration of the linguistic and semantic aspects of the instrument, and understanding of the context where the instrument was developed and where it is to be applied. A team, including the instrument’s developers, translators, and researchers is necessary to ensure a valid translation. This study suggests preliminary validity, reliability and acceptability evidence for the ACT when used with nurses in Swedish elder care.
Questionnaire; Translation; Validity; Health care context; Research utilization; Nursing
Healthcare-associated infections affect 10% of patients in Canadian acute-care hospitals and are significant and preventable causes of morbidity and mortality among hospitalized patients. Hand hygiene is among the simplest and most effective preventive measures to reduce these infections. However, compliance with hand hygiene among healthcare workers, specifically among physicians, is consistently suboptimal. We aim to first identify the barriers and enablers to physician hand hygiene compliance, and then to develop and pilot a theory-based knowledge translation intervention to increase physicians’ compliance with best hand hygiene practice.
The study consists of three phases. In Phase 1, we will identify barriers and enablers to hand hygiene compliance by physicians. This will include: key informant interviews with physicians and residents using a structured interview guide, informed by the Theoretical Domains Framework; nonparticipant observation of physician/resident hand hygiene audit sessions; and focus groups with hand hygiene experts. In Phase 2, we will conduct intervention mapping to develop a theory-based knowledge translation intervention to improve physician hand hygiene compliance. Finally, in Phase 3, we will pilot the knowledge translation intervention in four patient care units.
In this study, we will use a behavioural theory approach to obtain a better understanding of the barriers and enablers to physician hand hygiene compliance. This will provide a comprehensive framework on which to develop knowledge translation interventions that may be more successful in improving hand hygiene practice. Upon completion of this study, we will refine the piloted knowledge translation intervention so it can be tested in a multi-site cluster randomized controlled trial.
One of the most consistent findings from clinical and health services research is the failure to translate research into practice and policy. As a result of these evidence-practice and policy gaps, patients fail to benefit optimally from advances in healthcare and are exposed to unnecessary risks of iatrogenic harms, and healthcare systems are exposed to unnecessary expenditure resulting in significant opportunity costs. Over the last decade, there has been increasing international policy and research attention on how to reduce the evidence-practice and policy gap. In this paper, we summarise the current concepts and evidence to guide knowledge translation activities, defined as T2 research (the translation of new clinical knowledge into improved health). We structure the article around five key questions: what should be transferred; to whom should research knowledge be transferred; by whom should research knowledge be transferred; how should research knowledge be transferred; and, with what effect should research knowledge be transferred?
We suggest that the basic unit of knowledge translation should usually be up-to-date systematic reviews or other syntheses of research findings. Knowledge translators need to identify the key messages for different target audiences and to fashion these in language and knowledge translation products that are easily assimilated by different audiences. The relative importance of knowledge translation to different target audiences will vary by the type of research and appropriate endpoints of knowledge translation may vary across different stakeholder groups. There are a large number of planned knowledge translation models, derived from different disciplinary, contextual (i.e., setting), and target audience viewpoints. Most of these suggest that planned knowledge translation for healthcare professionals and consumers is more likely to be successful if the choice of knowledge translation strategy is informed by an assessment of the likely barriers and facilitators. Although our evidence on the likely effectiveness of different strategies to overcome specific barriers remains incomplete, there is a range of informative systematic reviews of interventions aimed at healthcare professionals and consumers (i.e., patients, family members, and informal carers) and of factors important to research use by policy makers.
There is a substantial (if incomplete) evidence base to guide choice of knowledge translation activities targeting healthcare professionals and consumers. The evidence base on the effects of different knowledge translation approaches targeting healthcare policy makers and senior managers is much weaker but there are a profusion of innovative approaches that warrant further evaluation.
There are few validated measures of organizational context and none that we located are parsimonious and address modifiable characteristics of context. The Alberta Context Tool (ACT) was developed to meet this need. The instrument assesses 8 dimensions of context, which comprise 10 concepts. The purpose of this paper is to report evidence to further the validity argument for ACT. The specific objectives of this paper are to: (1) examine the extent to which the 10 ACT concepts discriminate between patient care units and (2) identify variables that significantly contribute to between-unit variation for each of the 10 concepts.
859 professional nurses (844 valid responses) working in medical, surgical and critical care units of 8 Canadian pediatric hospitals completed the ACT. A random intercept, fixed effects hierarchical linear modeling (HLM) strategy was used to quantify and explain variance in the 10 ACT concepts to establish the ACT's ability to discriminate between units. We ran 40 models (a series of 4 models for each of the 10 concepts) in which we systematically assessed the unique contribution (i.e., error variance reduction) of different variables to between-unit variation. First, we constructed a null model in which we quantified the variance overall, in each of the concepts. Then we controlled for the contribution of individual level variables (Model 1). In Model 2, we assessed the contribution of practice specialty (medical, surgical, critical care) to variation since it was central to construction of the sampling frame for the study. Finally, we assessed the contribution of additional unit level variables (Model 3).
The null model (unadjusted baseline HLM model) established that there was significant variation between units in each of the 10 ACT concepts (i.e., discrimination between units). When we controlled for individual characteristics, significant variation in the 10 concepts remained. Assessment of the contribution of specialty to between-unit variation enabled us to explain more variance (1.19% to 16.73%) in 6 of the 10 ACT concepts. Finally, when we assessed the unique contribution of the unit level variables available to us, we were able to explain additional variance (15.91% to 73.25%) in 7 of the 10 ACT concepts.
The findings reported here represent the third published argument for validity of the ACT and adds to the evidence supporting its use to discriminate patient care units by all 10 contextual factors. We found evidence of relationships between a variety of individual and unit-level variables that explained much of this between-unit variation for each of the 10 ACT concepts. Future research will include examination of the relationships between the ACT's contextual factors and research utilization by nurses and ultimately the relationships between context, research utilization, and outcomes for patients.
In healthcare, a gap exists between what is known from research and what is practiced. Understanding this gap depends upon our ability to robustly measure research utilization.
The objectives of this systematic review were: to identify self-report measures of research utilization used in healthcare, and to assess the psychometric properties (acceptability, reliability, and validity) of these measures.
We conducted a systematic review of literature reporting use or development of self-report research utilization measures. Our search included: multiple databases, ancestry searches, and a hand search. Acceptability was assessed by examining time to complete the measure and missing data rates. Our approach to reliability and validity assessment followed that outlined in the Standards for Educational and Psychological Testing.
Of 42,770 titles screened, 97 original studies (108 articles) were included in this review. The 97 studies reported on the use or development of 60 unique self-report research utilization measures. Seven of the measures were assessed in more than one study. Study samples consisted of healthcare providers (92 studies) and healthcare decision makers (5 studies). No studies reported data on acceptability of the measures. Reliability was reported in 32 (33%) of the studies, representing 13 of the 60 measures. Internal consistency (Cronbach's Alpha) reliability was reported in 31 studies; values exceeded 0.70 in 29 studies. Test-retest reliability was reported in 3 studies with Pearson's r coefficients > 0.80. No validity information was reported for 12 of the 60 measures. The remaining 48 measures were classified into a three-level validity hierarchy according to the number of validity sources reported in 50% or more of the studies using the measure. Level one measures (n = 6) reported evidence from any three (out of four possible) Standards validity sources (which, in the case of single item measures, was all applicable validity sources). Level two measures (n = 16) had evidence from any two validity sources, and level three measures (n = 26) from only one validity source.
This review reveals significant underdevelopment in the measurement of research utilization. Substantial methodological advances with respect to construct clarity, use of research utilization and related theory, use of measurement theory, and psychometric assessment are required. Also needed are improved reporting practices and the adoption of a more contemporary view of validity (i.e., the Standards) in future research utilization measurement studies.
Organizational context has the potential to influence the use of new knowledge. However, despite advances in understanding the theoretical base of organizational context, its measurement has not been adequately addressed, limiting our ability to quantify and assess context in healthcare settings and thus, advance development of contextual interventions to improve patient care. We developed the Alberta Context Tool (the ACT) to address this concern. It consists of 58 items representing 10 modifiable contextual concepts. We reported the initial validation of the ACT in 2009. This paper presents the second stage of the psychometric validation of the ACT.
We used the Standards for Educational and Psychological Testing to frame our validity assessment. Data from 645 English speaking healthcare aides from 25 urban residential long-term care facilities (nursing homes) in the three Canadian Prairie Provinces were used for this stage of validation. In this stage we focused on: (1) advanced aspects of internal structure (e.g., confirmatory factor analysis) and (2) relations with other variables validity evidence. To assess reliability and validity of scores obtained using the ACT we conducted: Cronbach's alpha, confirmatory factor analysis, analysis of variance, and tests of association. We also assessed the performance of the ACT when individual responses were aggregated to the care unit level, because the instrument was developed to obtain unit-level scores of context.
Item-total correlations exceeded acceptable standards (> 0.3) for the majority of items (51 of 58). We ran three confirmatory factor models. Model 1 (all ACT items) displayed unacceptable fit overall and for five specific items (1 item on adequate space for resident care in the Organizational Slack-Space ACT concept and 4 items on use of electronic resources in the Structural and Electronic Resources ACT concept). This prompted specification of two additional models. Model 2 used the 7 scaled ACT concepts while Model 3 used the 3 count-based ACT concepts. Both models displayed substantially improved fit in comparison to Model 1. Cronbach's alpha for the 10 ACT concepts ranged from 0.37 to 0.92 with 2 concepts performing below the commonly accepted standard of 0.70. Bivariate associations between the ACT concepts and instrumental research utilization levels (which the ACT should predict) were statistically significant at the 5% level for 8 of the 10 ACT concepts. The majority (8/10) of the ACT concepts also showed a statistically significant trend of increasing mean scores when arrayed across the lowest to the highest levels of instrumental research use.
The validation process in this study demonstrated additional empirical support for construct validity of the ACT, when completed by healthcare aides in nursing homes. The overall pattern of the data was consistent with the structure hypothesized in the development of the ACT and supports the ACT as an appropriate measure for assessing organizational context in nursing homes. Caution should be applied in using the one space and four electronic resource items that displayed misfit in this study with healthcare aides until further assessments are made.
There is a lack of acceptable, reliable, and valid survey instruments to measure conceptual research utilization (CRU). In this study, we investigated the psychometric properties of a newly developed scale (the CRU Scale).
We used the Standards for Educational and Psychological Testing as a validation framework to assess four sources of validity evidence: content, response processes, internal structure, and relations to other variables. A panel of nine international research utilization experts performed a formal content validity assessment. To determine response process validity, we conducted a series of one-on-one scale administration sessions with 10 healthcare aides. Internal structure and relations to other variables validity was examined using CRU Scale response data from a sample of 707 healthcare aides working in 30 urban Canadian nursing homes. Principal components analysis and confirmatory factor analyses were conducted to determine internal structure. Relations to other variables were examined using: (1) bivariate correlations; (2) change in mean values of CRU with increasing levels of other kinds of research utilization; and (3) multivariate linear regression.
Content validity index scores for the five items ranged from 0.55 to 1.00. The principal components analysis predicted a 5-item 1-factor model. This was inconsistent with the findings from the confirmatory factor analysis, which showed best fit for a 4-item 1-factor model. Bivariate associations between CRU and other kinds of research utilization were statistically significant (p < 0.01) for the latent CRU scale score and all five CRU items. The CRU scale score was also shown to be significant predictor of overall research utilization in multivariate linear regression.
The CRU scale showed acceptable initial psychometric properties with respect to responses from healthcare aides in nursing homes. Based on our validity, reliability, and acceptability analyses, we recommend using a reduced (four-item) version of the CRU scale to yield sound assessments of CRU by healthcare aides. Refinement to the wording of one item is also needed. Planned future research will include: latent scale scoring, identification of variables that predict and are outcomes to conceptual research use, and longitudinal work to determine CRU Scale sensitivity to change.
Defining what constitutes a resident care unit in nursing home research is both a conceptual and practical challenge. The aim of this paper is to provide evidence in support of a definition of care unit in nursing homes by demonstrating: (1) its feasibility for use in data collection, (2) the acceptability of aggregating individual responses to the unit level, and (3) the benefit of including unit level data in explanatory models.
An observational study design was used. Research (project) managers, healthcare aides, care managers, nursing home administrators and directors of care from thirty-six nursing homes in the Canadian prairie provinces of Alberta, Saskatchewan and Manitoba provided data for the study. A definition of care unit was developed and applied in data collection and analyses. A debriefing session was held with research managers to investigate their experiences with using the care unit definition. In addition, survey responses from 1258 healthcare aides in 25 of the 36 nursing homes in the study, that had more than one care unit, were analyzed using a multi-level modeling approach. Trained field workers administered the Alberta Context Tool (ACT), a 58-item self-report survey reflecting 10 organizational context concepts, to healthcare aides using computer assisted personal interviews. To assess the appropriateness of obtaining unit level scores, we assessed aggregation statistics (ICC(1), ICC(2), η2, and ω2), and to assess the value of using the definition of unit in explanatory models, we performed multi-level modeling.
In 10 of the 36 nursing homes, the care unit definition developed was used to align the survey data (for analytic purposes) to specific care units as designated by our definition, from that reported by the facility administrator. The aggregation statistics supported aggregating the healthcare aide responses on the ACT to the realigned unit level. Findings from the multi-level modeling further supported unit level aggregation. A significantly higher percentage of variance was explained in the ACT concepts at the unit level compared to the individual and/or nursing home levels.
The statistical results support the use of our definition of care unit in nursing home research in the Canadian prairie provinces. Beyond research convenience however, the results also support the resident unit as an important Clinical Microsystem to which future interventions designed to improve resident quality of care and staff (healthcare aide) worklife should be targeted.
In the past forty years, many gains have been made in our understanding of the concept of research utilization. While numerous studies exist on professional nurses' use of research in practice, no attempt has been made to systematically evaluate and synthesize this body of literature with respect to the extent to which nurses use research in their clinical practice. The objective of this study was to systematically identify and analyze the available evidence related to the extent to which nurses use research findings in practice.
This study was a systematic review of published and grey literature. The search strategy included 13 online bibliographic databases: Cochrane Database of Systematic Reviews, Cochrane Central Register of Controlled Trials, MEDLINE, CINAHL, EMBASE, HAPI, Web of Science, SCOPUS, OCLC Papers First, OCLC WorldCat, ABI Inform, Sociological Abstracts, and Dissertation Abstracts. The inclusion criteria consisted of primary research reports that assess professional nurses' use of research in practice, written in the English or Scandinavian languages. Extent of research use was determined by assigning research use scores reported in each article to one of four quartiles: low, moderate-low, moderate-high, or high.
Following removal of duplicate citations, a total of 12,418 titles were identified through database searches, of which 133 articles were retrieved. Of the articles retrieved, 55 satisfied the inclusion criteria. The 55 final reports included cross-sectional/survey (n = 51) and quasi-experimental (n = 4) designs. A sensitivity analysis, comparing findings from all reports with those rated moderate (moderate-weak and moderate-strong) and strong quality, did not show significant differences. In a majority of the articles identified (n = 38, 69%), nurses reported moderate-high research use.
According to this review, nurses' reported use of research is moderate-high and has remained relatively consistent over time until the early 2000's. This finding, however, may paint an overly optimistic picture of the extent to which nurses use research in their practice given the methodological problems inherent in the majority of studies. There is a clear need for the development of standard measures of research use and robust well-designed studies examining nurses' use of research and its impact on patient outcomes. The relatively unchanged self-reports of moderate-high research use by nurses is troubling given that over 40 years have elapsed since the first studies in this review were conducted and the increasing emphasis in the past 15 years on evidence-based practice. More troubling is the absence of studies in which attempts are made to assess the effects of varying levels of research use on patient outcomes.
Interventions that have a better than random chance of increasing nurses' use of research are important to the delivery of quality patient care. However, few reports exist of successful research utilization in nursing interventions. Systematic identification and evaluation of individual characteristics associated with and predicting research utilization may inform the development of research utilization interventions.
To update the evidence published in a previous systematic review on individual characteristics influencing research utilization by nurses.
As part of a larger systematic review on research utilization instruments, 12 online bibliographic databases were searched. Hand searching of specialized journals and an ancestry search was also conducted. Randomized controlled trials, clinical trials, and observational study designs examining the association between individual characteristics and nurses' use of research were eligible for inclusion. Studies were limited to those published in the English, Danish, Swedish, and Norwegian languages. A vote counting approach to data synthesis was taken.
A total of 42,770 titles were identified, of which 501 were retrieved. Of these 501 articles, 45 satisfied our inclusion criteria. Articles assessed research utilization in general (n = 39) or kinds of research utilization (n = 6) using self-report survey measures. Individual nurse characteristics were classified according to six categories: beliefs and attitudes, involvement in research activities, information seeking, education, professional characteristics, and socio-demographic/socio-economic characteristics. A seventh category, critical thinking, emerged in studies examining kinds of research utilization. Positive relationships, at statistically significant levels, for general research utilization were found in four categories: beliefs and attitudes, information seeking, education, and professional characteristics. The only characteristic assessed in a sufficient number of studies and with consistent findings for the kinds of research utilization was attitude towards research; this characteristic had a positive association with instrumental and overall research utilization.
This review reinforced conclusions in the previous review with respect to positive relationships between general research utilization and: beliefs and attitudes, and current role. Furthermore, attending conferences/in-services, having a graduate degree in nursing, working in a specialty area, and job satisfaction were also identified as individual characteristics important to research utilization. While these findings hold promise as potential targets of future research utilization interventions, there were methodological problems inherent in many of the studies that necessitate their findings be replicated in further research using more robust study designs and multivariate assessment methods.
The Resident Assessment Instrument-Minimum Data Set (RAI-MDS) 2.0 is designed to collect the minimum amount of data to guide care planning and monitoring for residents in long-term care settings. These data have been used to compute indicators of care quality. Use of the quality indicators to inform quality improvement initiatives is contingent upon the validity and reliability of the indicators. The purpose of this review was to systematically examine published and grey research reports in order to assess the state of the science regarding the validity and reliability of the RAI-MDS 2.0 Quality Indicators (QIs).
We systematically reviewed the evidence for the validity and reliability of the RAI-MDS 2.0 QIs. A comprehensive literature search identified relevant original research published, in English, prior to December 2008. Fourteen articles and one report examining the validity and/or reliability of the RAI-MDS 2.0 QIs were included.
The studies fell into two broad categories, those that examined individual quality indicators and those that examined multiple indicators. All studies were conducted in the United States and included from one to a total of 209 facilities. The number of residents included in the studies ranged from 109 to 5758. One study conducted under research conditions examined 38 chronic care QIs, of which strong evidence for the validity of 12 of the QIs was found. In response to these findings, the 12 QIs were recommended for public reporting purposes. However, a number of observational studies (n = 13), conducted in "real world" conditions, have tested the validity and/or reliability of individual QIs, with mixed results. Ten QIs have been studied in this manner, including falls, depression, depression without treatment, urinary incontinence, urinary tract infections, weight loss, bedfast, restraint, pressure ulcer, and pain. These studies have revealed the potential for systematic bias in reporting, with under-reporting of some indicators and over-reporting of others.
Evidence for the reliability and validity of the RAI-MDS QIs remains inconclusive. The QIs provide a useful tool for quality monitoring and to inform quality improvement programs and initiatives. However, caution should be exercised when interpreting the QI results and other sources of evidence of the quality of care processes should be considered in conjunction with QI results.
The context of healthcare organizations such as hospitals is increasingly accepted as having the potential to influence the use of new knowledge. However, the mechanisms by which the organizational context influences evidence-based practices are not well understood. Current measures of organizational context lack a theory-informed approach, lack construct clarity and generally have modest psychometric properties. This paper presents the development and initial psychometric validation of the Alberta Context Tool (ACT), an eight dimension measure of organizational context for healthcare settings.
Three principles guided the development of the ACT: substantive theory, brevity, and modifiability. The Promoting Action on Research Implementation in Health Services (PARiHS) framework and related literature were used to guide selection of items in the ACT. The ACT was required to be brief enough to be tolerated in busy and resource stretched work settings and to assess concepts of organizational context that were potentially modifiable. The English version of the ACT was completed by 764 nurses (752 valid responses) working in seven Canadian pediatric care hospitals as part of its initial validation. Cronbach's alpha, exploratory factor analysis, analysis of variance, and tests of association were used to assess instrument reliability and validity.
Factor analysis indicated a 13-factor solution (accounting for 59.26% of the variance in 'organizational context'). The composition of the factors was similar to those originally conceptualized. Cronbach's alpha for the 13 factors ranged from .54 to .91 with 4 factors performing below the commonly accepted alpha cut off of .70. Bivariate associations between instrumental research utilization levels (which the ACT was developed to predict) and the ACT's 13 factors were statistically significant at the 5% level for 12 of the 13 factors. Each factor also showed a trend of increasing mean score ranging from the lowest level to the highest level of instrumental research use, indicating construct validity.
To date, no completely satisfactory measures of organizational context are available for use in healthcare. The ACT assesses several core domains to provide a comprehensive account of organizational context in healthcare settings. The tool's strengths are its brevity (allowing it to be completed in busy healthcare settings) and its focus on dimensions of organizational context that are modifiable. Refinements of the instrument for acute, long term care, and home care settings are ongoing.
While there is a growing awareness of the importance of organizational context (or the work environment/setting) to successful knowledge translation, and successful knowledge translation to better patient, provider (staff), and system outcomes, little empirical evidence supports these assumptions. Further, little is known about the factors that enhance knowledge translation and better outcomes in residential long-term care facilities, where care has been shown to be suboptimal. The project described in this protocol is one of the two main projects of the larger five-year Translating Research in Elder Care (TREC) program.
The purpose of this project is to establish the magnitude of the effect of organizational context on knowledge translation, and subsequently on resident, staff (unregulated, regulated, and managerial) and system outcomes in long-term care facilities in the three Canadian Prairie Provinces (Alberta, Saskatchewan, Manitoba).
This study protocol describes the details of a multi-level – including provinces, regions, facilities, units within facilities, and individuals who receive care (residents) or work (staff) in facilities – and longitudinal (five-year) research project. A stratified random sample of 36 residential long-term care facilities (30 urban and 6 rural) from the Canadian Prairie Provinces will comprise the sample. Caregivers and care managers within these facilities will be asked to complete the TREC survey – a suite of survey instruments designed to assess organizational context and related factors hypothesized to be important to successful knowledge translation and to achieving better resident, staff, and system outcomes. Facility and unit level data will be collected using standardized data collection forms, and resident outcomes using the Resident Assessment Instrument-Minimum Data Set version 2.0 instrument. A variety of analytic techniques will be employed including descriptive analyses, psychometric analyses, multi-level modeling, and mixed-method analyses.
Three key challenging areas associated with conducting this project are discussed: sampling, participant recruitment, and sample retention; survey administration (with unregulated caregivers); and the provision of a stable set of study definitions to guide the project.